j42

This is actually quite fascinating (and part of my upcoming book...)

Despite the obvious high/low-level language differentials, people have been managing multi-million-line C code-bases since the 80's. Then again, think of how little the build tooling has actually changed -- instead of fragmenting, you're left with llvm, gcc, make at the core of most compiled software.

JavaScript is the exact opposite with the lowest possible barrier to entry (built-in to every web browser...), and the proliferation of frameworks and libraries may be due to this in combination with the lack of fundamental understanding of design patterns that can scale. Lots of people trying to partially solve symptoms, missing the forest for the trees.

Theoretically there's nothing preventing good patterns in high-level UI development, but I'm not quite sure it's been done right, yet and have no idea when the dust will settle.

Oh it absolutely is, and I'm on your side.

My point though, is that it's always a "secondary" or "tertiary" business concern... a point emboldened by the number and frequency of data breaches -- always followed of course, by the email newsletter follow-up & mea culpa. "We care about the security of your data, we swear. We regret to inform you that ..."

Sadly things are not always as they should be, friend...

Probably because very little innovation happens to any long-standing protocol until it becomes a primary business concern.

Almost every business relies on OpenSSL or some equivalent, but how many actually learn enough about SSL to contribute back to the codebase? Not many, because despite the need there's little acclaim or funding to be had pursuing things that won't make direct revenues, regardless of their importance.

Protocols like this generally don't get updated until it becomes a matter of necessity -- either by public awareness (we're far from that point) or someone designing the "next big thing" needs an un-implemented feature and contributes.

I do think there's some value to pointing out the irony though; two men from two cultures that couldn't be more distinct, with titles that could be synonyms, using what is essentially the same scripted prose to control and appease the media/populace.

Personally I find it disheartening. I think the Chinese are copying key aspects of the American Media/propaganda model because it's been demonstrated as so damn effective...

You seem extremely knowledgeable about high-performance servers!

Your example is fantastic -- I realize it's written in C, but perhaps you could explain how in the context of a tool like this (current article) might allow it to function as an end-to-end webserver?

What kind of limitations? What are the advantages, security implications? What is it well-suited for?

Really appreciate a reply -- I'm writing a book on architecture design and I'd really love to include some elements from the bleeding-edge of performance -- you clearly know what you're talking about.

Can you tell me where I may find more info on KDB or how this may compare to more traditional serving options for static resources, HLLs or anything that might fit well in a large tree.

I'm trying to understand exactly what its limitations are though, both in terms of what it can do and when it may be a poor candidate (when lower req/s is ok and greater flexibility is needed).

Hah, the way you phrase that makes me wonder what you found.

Right now really just web apps. Content sites are doable, and I know people running networks, but it's not possible to do profitably without clickbaiting, low-quality mass-produced content, and other greyhat techniques I don't particularly like.

I think it's fair to show you an ad if I save you 30 seconds to a minute of time. Creating garbage content slideshows with writers from the 3rd world countries to rack up pageviews is one step over the line for me...

Simple apps are great because they have utility (so google always approves your tag), and honestly the doubleclick exchange isn't so bad -- with enough simple single-page apps, it accrues pretty quickly.

Not exactly, I think it was a poor choice of words as I was referring to my build/deployment tooling.

If I have an idea (e.g. a "ES2015 to ECMA6" converter), I'd look for a library to do this (if there isn't an open source one, I'd consider building my own as a learning experience).

Setting up the webpage up (converter/form) is trivial since it's really just one page and a few settings.

The really cool part is I've built laravel commands to do the hard work for me. They can:

- buy domains based on a keyword search with some common regex-like patterns

- set up a git hook to continuously deploy to an AWS micro instance via codedeploy/CircleCI

- manage different adsense codes across properties

So, if you're wondering these are things that have utility. I'm optimizing the pains of setting up each site, though.

I do build adservers for a living, but never once have I created a crappy content-farm click-baiting blog -- they are the scourge of the internet >:{

[edit] lastly, I'm not sure if you have a problem with google ads (doubleclick/google publisher network) but I think they're pretty sensible in what they allow... no reasonable person would take issue with that.

I have two books in my queue actually, and this is one of them.

There are parts of my content-property model I'm not ready to share, but most of it has been structured into a cohesive framework of how I:

1. find niches 2. automate site creation 3. optimize w/ gpt (google publisher tag)

I'm sorry to say I won't be releasing that first... The first is due within the next 12-14 days, after which it should take me 1-2 months to finish writing & editing the adsense-property-model guide.

To leave you with something tangible, it's very feasible to produce 5-figures per day with only hosting cost as your overhead ($300/mo).

[edit] without automation tools though, it would obviously be very tedious to produce the number of properties required to have that kind of income by hand. process flow & site generation/management is the crux of this strategy.

Well, "startup" is a poor choice of words, but he actually has a half-decent idea.

As someone who owns dozens of little "tool" sites (think less/scss converters, meme generators, JS beautifiers, etc), I can tell you each is probably 1-2 pages, took an hour to build and thankfully due to some domain squatting (kw in domain) and a low bounce rate I don't have to worry much about SEO.

As for the Adsense revenue, I think you'd be quite surprised.

One is an afternoon, not a startup idea.

50, on the other hand, could be passive income for a very long time.

Just something to consider before jumping to negativity. ;)

It's really interesting to see services like this emerge.

In my opinion, the merchant processing/gateway revolution happened when providers (Stripe, Braintree, et al) started providing quality APIs for user/profile/subscription management and took the burdens of PCI compliance off of the companies building consumer products.

On the surface, I feel like this is a great way to offload the liability of storing sensitive user data -- though it also creates a central source of failure. Success is predicated on Luno securing their data; if they can't, the model would die.

If they can, it's possible we'll start to see a mass-migration of authentication-based apps switching to these service, if only for the legal intention of offloading liability.

Really, a fascinating model.

You may have misunderstood me there.

Yes, Islam is a particularly violent religion -- if you have read the Koran, many of its concepts seem very incompatible with the idea of a free secular society. Then again, the bible has the crusades.

I'm personally an atheist and I honestly don't think the problem is the the text itself but the cultural, ideological conflicts of an impoverished region that allows whoever "shouts the loudest" to assume power. The kids committing these atrocities probably couldn't even tell you what you just told me about Mohammad -- my point is that they are brainwashed and utterly uneducated so whomever comes along and says "this is god's word" is who they will listen to.

Anger + Desperation - Education = Extremism

You won't solve this problem by banning the Koran, but if you can get Muslims everywhere to renounce this "us-or-them" culture in favor of a more moderate interpretation (you know, how all religions seem to evolve if they want to survive) then perhaps we can neuter these kinds of groups before there's a power-vacuum?

It does if you play devil's advocate and follow the chain of reasoning.

If god is great and non-believers are bad and "god" says it's righteous and just to punish the non-believers, then naturally doing "god's work" is doing no harm?

Actually the truth is even messier... most of these young men committing atrocities are merely indoctrinated pawns who know very little of their own religion and instead defer to their "emir."

This ideological poison is being propagated by those individuals, with power/financial interests back in the middle east. I think the individual committing the act believes they are doing good, and the individual who convinced them to do it is too morally corrupted & detached to care about ideals such as "civilian life."

There are far more atrocities that occur in this world than there are psychopaths in the general population.

If we hope to make any progress toward peace, I think we need to truly understand the reasons why and how weak, impressionable minds with poor cultural integration can be manipulated to commit such atrocities.

It's easy to label these individuals as determined, unreachable psychopaths (particularly out of fear) but the sad truth is, most extremism is borne not of evil but of weakness. A select few manipulate this weakness to convince otherwise insignificant people, often with desires of grandeur to commit unthinkable acts. This power of perspective becomes increasingly obvious as you realize most problems with immigrants in European countries occur in the 2nd and 3rd generations -- those who have seen the true horrors of war first hand are not so easily fooled.

The hard truth is: if society doesn't provide susceptible minds with alternatives first, a small but steady % will be at the mercy of whomever comes along promising "answers."

As someone who develops high-performance systems and is continually learning, I think this is a fantastic idea!

Honestly, something that seems desperately needed as that knowledge is currently spread out among hundreds of thousands of blog posts, forums and threads -- diamonds in the rough.

I'm going to try to get something published on gumroad (and open-sourced on github) in this vein, if you're interested let me know and I'll reach out when it's done :)

Because it has never been a question of who is deserving of what...

The poor have no leverage, and realistically, no one is going to give that to them.

We're not talking about an idealistic re-working of civic values, but the natural tension that already exists between private interests and the public sector.

One need only look to antitrust law to see why letting any single locus of control grow unchecked can wreak havoc on a market/economy -- I think we should appreciate the few checks & balances we have left, lest this notion of naive idealistic equality allows them to slip away unchallenged.

I was actually planning on testing a 33% deployment this evening, I guess what I'm really wondering is that because this is a low-level networking adjustment that modifies locking behavior, are there 'gotchas' I should be aware of beforehand?

I'll test and report back regardless; only afraid of the scenario where the test goes well, and the entire network goes down a week later (e.g., a wonderfully-fun time we had previously chasing down rogue epoll queues and zombie processes that only occurred after a threshold of sustained load).

Can anyone comment on whether it would be a good or bad idea to try to implement this in production ASAP?

I'm a bit far removed from the Linux kernel to be comfortable auditing that myself, but I run some high-volume/low-latency exchange clusters and the limiting bottleneck on requests/box has always been due to SYN/ACK negotiation.

I solved that currently by having hundreds of smaller servers, which distributes the network load quite nicely but isn't even coming close to maximal utilization of CPU. Realistically since moving from PHP to Haskell we're seeing about 1k req/s/box, but without the network slowing things down we're looking at a magnitude of increase on the current hardware.

Just FYI I've already handled the obvious, such as intelligent caching, nginx split upstreams, TIME_WAIT and reuse adjustments (1s), et al. Qualitatively, we're looking for an assurance on <= 100ms TTFB for the 99th percentile, in a way that allows us to use the most of our hardware via green threads.

It's because there's at least one person on-board the company who understands the technical underpinnings, and has incentives that (usually) align with shareholders -- versus consultants whose bottom-line is never entirely dependent on the success of your company.

I'm not sure I see that as a fatal flaw... ultimately, the user is sitting in front of a digital box that can guide and prompt them in all the same ways a researcher can--the only limit is in performing tests that require a medical professional to assess biomarkers. If the pace of medical device development continues, it's even reasonable to think something like a Theranos-that-works could commoditize the process while being intrinsically tamper-resistant.

Regardless, users can be prompted to perform any software action (knowingly or unknowingly, to affect bias) and that action can be measured by the system. It may so happen that every critical measurement occurs unbeknownst to the user, before they self-report anything (if at all). As we are currently undergoing a period of sensor-proliferation (fitness/health devices, wearables, internet of things, etc...) it's not unrealistic to think we will soon be able to instantly correlate data from a smartphone camera, blood/tissue, and the cloud.

Now there's always the problem of intentional fraud/deception, but I think the aggregate nature solves that problem. A small percentage will try to "break" the system, and that small percentage will never surpass a critical threshold with enough volume. In terms of ML/SVM's, we're now very good about filtering outliers or "misrepresented data"... while the responsibility is on you to develop a reliable classifier (for data-consistency more than arbitrary measurement), I imagine at scale you could infer trends with the same relative accuracy of traditional academia and research.

It's a really fascinating new direction--even if only an adjunct to traditional research--and I'll definitely be keeping an eye on the project.

I'm afraid you're misinformed, but for socially-conscious individuals (ie, they don't support a scorched-earth, ends-justify-the-means policy to get that $250,000* from your nephew for torrenting Inside Out) I'm including my comment below.

If you are a programmer, security researcher, artist, or entrepreneur you can make a difference.

1. As an individual: if you understand the methods, contribute to open-source tools that allow individuals to exercise their rights. https://github.com/apprenticeharper/DeDRM_tools (one example)

Artists: Use self-publishing platforms (gumroad, bandcamp, even spotify...) and self-incorporate. Discriminate against giving your business to companies that don't support open, sane protocols. Don't let them exert their power against the populace through backdoor trade deals.

Entrepreneurs: Create new content delivery and streaming platforms that force the transition to digital--rightsholders like to claim that piracy is responsible for their failed economics, though the truth is that they had an artificial market advantage of scarcity. User-generated content has bloomed with the advent of digital, and more consumer choice is a death knell to the traditional monopoly.

2. As a cause: support the EFF, and any politician looking to work with the FCC who understands this issue is deeper than "restricting content," and could undermine the rights of property and security of ownership. Do not trust anyone who does not comprehend the societal implications of critical infrastructure being "security through obscurity." http://apps.fcc.gov/ecfs/comment/view?id=60001303221

The security of your laptop, the concept of personal ownership, and your right not to be digitally inspected at over 40 international borders is at stake.

* figure revised to more accurately reflect the reality of the american justice system.

This.

Everyone should take note of the proper way to fight encroaching copyright law:

If you are a programmer, security researcher, artist, or entrepreneur you can make a difference.

1. As an individual: if you understand the methods, contribute to open-source tools that allow individuals to exercise their rights. https://github.com/apprenticeharper/DeDRM_tools (one example)

Artists: Use self-publishing platforms (gumroad, bandcamp, even spotify...) and self-incorporate. Discriminate against giving your business to companies that don't support open, sane protocols. Don't let them exert their power against the populace through backdoor trade deals.

Entrepreneurs: Create new content delivery and streaming platforms that force the transition to digital--rightsholders like to claim that piracy is responsible for their failed economics, though the truth is that they had an artificial market advantage of scarcity. User-generated content has bloomed with the advent of digital, and more consumer choice is a death knell to the traditional monopoly.

2. As a cause: support the EFF, and any politician looking to work with the FCC who understands this issue is deeper than "restricting content," and could undermine the rights of property and security of ownership. Do not trust anyone who does not comprehend the societal implications of critical infrastructure being "security through obscurity." http://apps.fcc.gov/ecfs/comment/view?id=60001303221

The security of your laptop, the concept of personal ownership, and your right not to be digitally inspected at over 40 international borders is at stake.

I'm truly confused by your message, because your latter statement sounds as if you are in agreement?

No one (intelligent) is trying to make a moral argument for 'piracy' or media entitlement, but rather saying that there do exist circumstances where the letter of the law deserves to be ignored as it runs contradictory to the spirit.

The spirit of copyright law is to ensure rights-holders are fairly compensated, and the unfortunate confluence of many complex factors has precluded this.

Rights-holders are understandably scared of technology's ability to level the playing field (by increasing access and decreasing their exclusivity advantage), and thus far most have chosen the historically-impotent strategy of hardline enforcement over adapting services and creating new revenue streams. What bothers me is that they hold the artists out to the public and say "look at this poor starving fella," meanwhile no one has any idea that their new streaming-media licensing agreement entitles artists to ~2-5% of the total earnings generated.

I'm particularly sensitive to this issue because I work in ad-tech. People (myself ironically included) love ad blockers, and I'd argue it's for good reason. Unless the implicit contract between those monetizing and those consuming is respected, everyone loses in the arms race that follows. In our industry it's been adapt-or-die (create products that don't hurt the user's experience), and that's the way it should be. Thankfully nobody is lobbying in congress to stipulate how you may use your eyeballs.

Hollywood and Telecom have historically received unprecedented favoritism in this country, and it's possible we're all on the verge of paying the price. It will only continue to encroach upon our individual rights as society becomes increasingly digital.

I don't think anyone is entitled to anything for free, but I'm a realist and a pragmatist. I will reverse engineer and circumvent the things people say I can't until the day I die :)

Actually, piracy is the act of forcefully commandeering another's maritime vessel; theft is the legal act of taking & depriving another of property, and I'm really not sure what "fucking" has to do with it at all.

Piracy is the correcting hand of the free market, where obtaining things for free is easier and less byzantine than by paid channels. Psychological research has proven time and time again that most people are willing to pay, but unwilling to have their personal rights trampled by draconian licensing and DRM.

Luckily your opinions, however misinformed, are irrelevant because anyone who understands this will never give up the fight. We understand how international trade deals and copyright law are being used offensively against the public, and we will not relinquish control over the devices we've rightfully purchased.

I'd suggest you get used to it. We're here to stay.

I'm the original poster, and I was talking about ads, not metered queries. :)

[edit] and perhaps that was a poor choice of words. as a developer of high-volume low-latency exchange feeds, "query" has a bit of a different meaning to me.

I'm sorry but I'm not quite sure what you mean by this?

In 2011, 96% of Google's revenue was from AdWords -- in recent years that has lowered slightly due to their diversification and cloud services, but it's still at least a 70% share of their gross revenues.

Facebook is currently ~80%.

The money is in the ads.

While not stated eloquently, I do think there's some truth in that.

1. Open API's, good documentation and community engagement goes a long way in establishing developer trust. I don't believe developer trust is correlated significantly with revenue (directly), but it's absolutely correlated with acquisition. I'd argue that Angular or React were partially successful because of the stickiness of their communities, and that's a powerful self-perpetuating force.

2. Monetization is in the queries. Facebook and Google know this. Throwing aside all of their auxiliary services, emerging markets, hardware, et al... between 70-90% of their respective revenue is from direct advertising. If there's a way to scale or "ramp" their revenue model, this would be the portion to focus on.

Really, though?

I always design with accessibility in mind, but I feel like there are more visceral complications than having to add a little extra context to a web search.

Almost everyone in development has encountered unfortunate name collisions that make what you're looking for hard to find... it's a pretty common inconvenience. Usually you just refine the query, like: "JAWS screenreader." The first result takes you where you probably intended to go, and it's at most 1-2 keystrokes away.

I think the real lesson here is to design inclusively. HTML has a lot of handy attributes to augment content; knowing & implementing the essentials means someone can follow the core flow of your site/application without relying on visual cues. And for the pragmatists... it has the added benefit of self-documenting and increasing semantic conformity in your markup.

Hey, thank you for this!

Bought your book a while ago -- didn't realize your background was in ad tech. I personally eeked out ~30ms avg latencies with PHP by modifying ReactPHP (non-blocking event loop) and getting creative with Redis and nginx.

It's performing comparably at high loads, and saving us a ton of money but obviously... it's PHP. I'm obsessed with Haskell and have spent a bit of time playing around with HLearn as I think SVM is an obvious use case where it would excel (though in terseness/structure more than performance relative to C).

Any advice on selling a rewrite and/or challenges as one nears 1B daily queries for which Haskell would be uniquely advantageous?

I want to integrate Haskell in our high-frequency serving arm (or as a backup, for machine learning), but honestly don't know enough about it relative to performance to sell its advantages over Erlang, Go or even C. Aside from the obvious "functional is better" and "types are great," which tend to appeal to the folks who see beauty in the language, is it feasible to claim that Haskell will result in a faster, more performance program in less time? ... And not to wear out my welcome, but I'm interested in reading more about performant Haskell (as per your post); any resources you recommend?

It comes across as a brilliantly absurd joke at first, though I suspect that to be a case of poor translation and an even poorer choice of analogy.

I see this pertaining to quantum physics/computing. I think it's trying to posit: "fluctuations in physically-coupled systems (tug-of-war dynamics) can be used to reliably and efficiently infer state/logic when the initial, discrete distribution is known."

To break that down a bit...

- They describe tug-of-war dynamics as a sort of "rigid physical coupling," and briefly allude to practical applications, stating that they've implemented this dynamic in "... quantum dots, single photons and atomic switches."

- They are using a poor example as the conceptual nature of the 'slot machines' and iron bar seems irrelevant -- it's just an analogy for a "measurable state" represented in a physically-coupled construct.

As for why it's relevant, it seems to be the cornerstone of an alternate paradigm to computing that when scaled down (to nanotechnology and quantum particles) can allow physical systems to represent logic and state without electrical transistors. That would appear to have application in every field from medicine to defense (smart materials, targeted drug delivery, et. al).

The breakthrough has 0 to do with the lovely "iron bar," and instead seems to be in their method of physical coupling, allowing them to represent binary switches reliably in the physical realm at quantum scale.

Then again...

"Other than this fluctuation, we added another fluctuation to our model. The important point is ... fluctuations."

http://i.imgur.com/bHFwiof.jpg

This actually is within the realm of common sense and inductive/deductive reasoning if you know a bit about nutrition.

First I'll agree with you that most people lack self control, and that the increase in portion sizes is a significant factor contributing to the obesity epidemic we're seeing right now.

That said, as someone who's been tangentially involved in product engineering for years, literally everything from the packaging to the chemical composition is designed the re-enforce addictive behavior (which is intrinsic to human psychology). Serving sizes are also often deceptive as they don't clearly indicate the macronutrient content for an entire package (as they do in many other countries). Sure, 7g of sugar per serving seems ok for cookies, until you realize that's every 2 in a box of 20 and you've eaten the whole thing (70g).

Most preservatives and artificial sweeteners are actually fairly inane, despite scary-looking ingredients lists. What you should be worried about are the cheap forms of sugar, refined carbohydrates, hydrolyzed oils/trans fats (which are thankfully declining in popularity), and HFCS/cane-sugar-based drinks which average 32-68g of sugar per bottle.

All of these things cause a condition called insulin resistance, which leads to obesity and eventually if untreated, acquired diabetes. Insulin is the hormone that extracts glucose (energy) from carbohydrates (food), and turns it into a source of energy for your cells. [It is well established scientifically that the aforementioned "watch-list" all spike your insulin levels, and that instability in your insulin levels relative to blood glucose is bad.](http://www.ncbi.nlm.nih.gov/pubmed/2995635)

In layman's terms, these things make your body release disproportionate amounts of insulin compared to the amount of food you've ingested, and this has a few downstream effects; a large spike can make you feel intensely hungry (why people say "empty" foods make you eat more than you would otherwise), and most importantly insulin becomes less effective at transporting glucose. That's the beginning sign of Insulin Resistance Syndrome, and its correlation to obesity is well established.

Ultimately it's because of the composition of processed foods, not any dangerous chemical additive (with a few notable exceptions) that most long-term complications arise.

Looking back 30 years you may be correct, but looking back 500 to pre-industrial diets, you'll notice sugar was less prevalent in the food of most Anthropocenes, and fiber was much more common. The sugar lobby of the 1900's is largely responsible for its GRAS status and current abundance.

These days we mill our grains into cereals, removing any traces of fiber/micronutrients and inject large amounts of liquid sugar into most beverages for 'flavor,' having the compound effect of up-regulating what we perceive to be "sweet." Both of these things make our bodies release far more insulin than is needed to extract their energy, making us hungrier disproportionally to caloric intake and setting the stage for a diabetic future.

To the unconscious consumer, these forces are silent and automatic. For as little self control as people may have, I personally can't place sole blame on the individual when I know the entire chain of supply to be rigged against their nutritional best interests.