Node.js in Flame Graphs

http://techblog.netflix.com/2014/11/nodejs-in-flames.html

by stoey • 12 years ago

778 240 12 years ago

techblog.netflix.com

gadders5• 12 years ago

OFFTOPIC: "Today, I want to share some recent learnings from performance tuning this new application stack."

The word you want is "lessons".

quarterto3• 12 years ago

"learnings" is a perfectly cromulent word: https://books.google.com/ngrams/graph?content=learnings&year...

gadders5• 12 years ago

Nope. You see an option for a plural here? http://www.merriam-webster.com/dictionary/learning

mcguire• 12 years ago

If you are Sacha Baron Cohen, yes.

mikeryan2• 12 years ago

OFFTOPIC: "Today, I want to share some recent learnings from performance tuning this new application stack."

The word you want is "lessons".

Jeez, not only is your comment offtopic and pedantic, it's also wrong.

swasheck• 12 years ago

"it's also wrong."

I believe the word you want is "incorrect."

/sarcasm

gadders5• 12 years ago

I flagged it as Offtopic. But it is correct. "Learnings" is at best a pointless affectation, like saying "utilised" instead of "used". At worst, according to the OED, it's not even a word.

http://english.stackexchange.com/questions/19227/plural-of-l...

Also: http://www.merriam-webster.com/dictionary/learning No plural option.

jwhitlark• 12 years ago

I didn't like it when people started using "learnings", but I'm beginning to appreciate some subtle shadings it seems to have over lessons. It seems more discovery oriented, and less sure it's correct, or at least, less widely correct. Sort of like "hacky, working hypotheses, that we think are worth sharing."

ajsharma• 12 years ago

This is the first I've heard of restify, but it seems like a useful framework for the main focus of most Node developers I know, which is to replace an API rather than a web application.

ecaron2• 12 years ago

You're going to love restify. We use it at TrackIf and can't imagine our API running on anything else. Couple it with Swagger and you won't be looking back :-)

vkjv• 12 years ago

...as well as increasing the Node.js heap size to 32Gb.
...also saw that the process’s heap size stayed fairly constant at around 1.2 Gb.

This is because 1.2 GB is the max allowed heap size in v8. Increasing beyond this value has no effect.

...It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers.

It it is non-trivial (not possible?) to do this in O(1) for routes that use matching / wildcards, etc. This optimization would only be possible for simple routes.

quarterto3• 12 years ago

There are ways around that: http://blog.caustik.com/2012/04/11/escape-the-1-4gb-v8-heap-...

herge2• 12 years ago

It it is non-trivial (not possible?) to do this in O(1) for routes that use matching / wildcards

I'd be impressed if they did it consistently in O(1) for static routes. I think they were looking for O(log(number of different routes)) instead of O(n).

azinman2• 12 years ago

If the paths have some kind of non crazy Regex leading up to what gets munched for path variables (eg /path1 versus /path2) then you could at least build a tree of maps at each level, which would be roughly constant time for the most common cases.

tedchs• 12 years ago

That seems like a pretty low size to me... how are people getting around this when they need to handle >1.2GB of data on Node?

jonny_eh3• 12 years ago

Native code modules I assume.

thedufer8• 12 years ago

It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers.

Its actually quite clear - most routes are defined by a regex rather than a string, so there is no built-in structure (if there's a way at all) to do O(1) lookups in the routing table. A router that only allowed string route definitions would be faster but far less useful.

I can't explain away the recursion, though. That seems wholly unnecessary.

Edit: Actually, I figured that out, too. You can put middleware in a router so it only runs on certain URL patterns. The only difference between a normal route handler and a middleware function is that a middleware function uses the third argument (an optional callback) and calls it when done to allow the route matcher to continue through the routes array. This can be asynchronous (thus the callback), so the router has to recurse through the routes array instead of looping.

cataphract• 12 years ago

On the other hand, without the recursion I don't see how the flame graph would have been helpful.

msmillie• 12 years ago

In flame graphs heavy iteration tends to give "wider" structures. Recall the stacks aren't ordered on the X axis by time, but by contents.

quotemstr3• 12 years ago

Of course there's a faster way! Combine all the routes into a DFA, then run the DFA over the URL. It's guaranteed to run in constant space and O(n) (n=URL length) time! The union of any set of regular languages is itself a regular language.

You can use Ragel[1] to build your automaton.

[1] http://www.colm.net/open-source/ragel/

thedufer8• 12 years ago

Yeah, but what's n there? Isn't that going to be something like the sum of each route's length? That doesn't buy us anything (well, probably a smaller constant).

Unless you go the NFA route, but I'm pretty sure that costs non-constant space.

badgersandjam2• 12 years ago

This. I wrote something that did this a few years ago. It took n patterns (not regex, simpler) and turned them into one DFA state table with a function ptr stored for each final state. Then it had a tiny runtime. Never thought of using it for route tables however.

Nice idea.

Edit: I did consider rewriting the runtime as a forth style interpreter as well but the time never appeared.

ChuckMcM4• 12 years ago

Oh that is very cool, I am definitely going to have to spend some time playing with that.

andrewstuart22• 12 years ago

This approach will of course work, but you can't have a middleware stack with a defined order using that approach, unless I'm mistaken.

Sure, you could use all routes that match, but is there a way to specify the order for all handlers, and whether or not you should continue after one handler is done?

rtpg• 12 years ago

This is the reason we need more people in JS land taking compiler courses and the like. There's a lot of tech out there that are speed sensitive yet do not apply the "best" solution to a problem solved in the 70s.

koenigdavidmj2• 12 years ago

There is a library (I think it is Google's re2) that supports throwing a bunch of regular expressions into a single structure, matching an input string against all of them at once, and answering which patterns matched the input string. This gets you route lookup in time linear to the input string (or if not linear, still better than checking all the patterns one after another).

f2f• 12 years ago

re2 is described here: http://swtch.com/~rsc/regexp/regexp3.html

rudolf02• 12 years ago

I use RE2 and currently achieve this with regular string concatenation. Do you know if they provide an API for doing this without concatenation, and if it would be faster?

ksherlock• 12 years ago

You could compile all the regular expressions into a table. Lex has been doing that since, oh, 1975.

andrewvc7• 12 years ago

A lot of people here are right, the right way is with an NFA. I just want to add that the solution is not even hard, you can do it with string concatenation and capture groups using regexps. Regexps are NFAs, and are highly optimized C code in just about every JS engine.

If I have the routes /foo/bar and /foo/bar/(\d+) I can generate the regexp ((^\/foo\/bar$)|(^\/foo\/bar\/\d+$))

I'm not at all surprised, the quality of software in node is pretty low, I've seen numerous issues in node libs being just as boneheaded. I swear, the fact that the express devs overlooked a key optimization is crazy. Rails, by way of example, uses the Journey engine to solve this problem (https://github.com/rails/journey)

llimllib• 12 years ago

Regexps with backtracks are not regular languages (IIRC, right?), does that matter here?

mjs• 12 years ago

It looks like routes are either strings (exact match) or regular expression objects. I don't think there's a way to combine regular expressions in JavaScript.

I don't think this was an unreasonable implementation. Sure, it could be optimized, but no-one needed it enough to do the work. (Even Netflix didn't need a faster matcher, they needed a matcher that didn't slow down between the 1st match and the nth.)

thedufer8• 12 years ago

I can generate the regexp ((^\/foo\/bar$)|(^\/foo\/bar\/\d+$))

And how would you know which one got matched? The regex match isn't going to tell you that. Also, it needs to recognize if multiple were matched, which is definitely not going to be done by the built-in regex matcher.

It's certainly possible, but pretending it's trivial isn't helping, either.

plorkyeran• 12 years ago

Given that combined regex, how do you determine which of the routes that went in to it was the one that matched?

spion3• 12 years ago

This indicates it probably wont be much faster in JS

http://jsperf.com/regex-list-vs-full

Most of the time in the single-regex case is likely spent on allocating the matches array (which for a router with 300 routes would have over 300 undefined elements on every execution)

V8 is not quite the typical dynamic language runtime. Code that would be dead slow in others is optimized really well by the amazing JIT.

LunaSea• 12 years ago

Oh yes, I forgot about the legendary Ruby On Rails code quality: http://puppetlabs.com/security/cve/cve-2013-0277

andrewstuart22• 12 years ago

The other thing you can't properly get using a map is precedence. By building an array you can go through it in order so that middleware is run in the order it was defined in the code.

That way, if you have any route definitions that match early on, especially if they allow execution to continue ( by calling next() ), they will run before any following code does.

This allows your code to be very expressive and intuitive when reading it, as things have a defined order.

I imagine the recursion is also done so that you can use a route object anywhere in that ordered stack and it will apply all its own handlers, in the order they were defined in the code.

cdoxsey• 12 years ago

Most routes are prefix matches, so a trie (or one of its variants) would work well.

debacle• 12 years ago

Doesn't this seem like a bug in the express router? All of the additional routes in the array are dead (can't be routed to).

dugmartin• 12 years ago

No, because the route list is also used for middleware. The recursive search is because middleware routes have a third next parameter that allows the search to run asynchronously.

ecaron2• 12 years ago

My biggest takeaway from this article is that Netflix is moving from Express to Restify, and I look forward to watching the massive uptick this has on https://github.com/mcavage/node-restify/graphs/contributors

sadkingbilly• 12 years ago

Yes, but their original bug was from dynamically loading routes from an external source. I don't see how Express is to blame for this. Moving to Restify is not a solution, but they state having different reasons for moving (support for bunyan logging? But Express already supports this too).

ecnahc515• 12 years ago

the bug was related to dynamically loading routes, but the true cause was that express allowed duplicate handlers. They were loading routes dynamically correctly, that wasn't a problem, it was that when doing that express let them duplicate routes.

gaving• 12 years ago

not exactly highly endorsed https://twitter.com/tjholowaychuk/status/382084838179614721

dagw• 12 years ago

While I kind of agree the restify gives you "almost nothing" when compared to some larger frameworks, in my experience "almost nothing" is basically all I need from my framework when writing REST APIs and restify has had me covered every time.

TheLoneWolfling6• 12 years ago

benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time

1ms / entry? What is it doing that it's spending 3 million cycles on a single path check?

jdmichal• 12 years ago

Running (uncompiled?) regular expressions, it seems.

baudehlo3• 12 years ago

V8 compiles regular expressions to machine code.

mikeryan2• 12 years ago

So I was a bit unclear on the parent's post but I don't think this time was on a route lookup if I'm reading the thread and post correctly the static file handler getting inserted multiple times. This handler will generally match on any route but then is doing something like "if file exists, return static file, if not look for the next handler" in this case the "if file exists" part was the "path check" thats taking 1 ms and was happening multiple times.

I could be wrong but it seems like the design of the route lookup mechanism (the global array) was actually a bit of a red herring, the real issue was the ability to attach multiple instances of the same handler to the same route.

Something was adding the same Express.js provided static route handler 10 times an hour. Further benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time.

forrestthewoods• 12 years ago

If I had to pick one line to highlight (not to criticize, but was a wise lesson worth sharing) it would be this one:

"First, we need to fully understand our dependencies before putting them into production."

gdulli• 12 years ago

In my experience developers constantly overestimate the gain of using a new dependency and underestimate the amount of effort it will take to sufficiently understand it. (Or fail to make the effort, not understanding the risks.)

This is why developers without significant experience should not be making decisions about the tech stack.

_RPM• 12 years ago

[Subjective] Not to criticize the Express.js code base, but have you tried reading it? It is very complicated and there are a bunch of clever things going on. I think it could have been written simpler and easier to understand. A problem with frameworks is that they are written by people who want to show off how clever they are.

drderidder• 12 years ago

  > our misuse of the Express.js API was the 
  > ultimate root cause of our performance issue

That's unfortunate. Restify is a nice framework too, but mistakes can be made with any of them. Strongloop has a post comparing Express, Restify, hapi and LoopBack for building REST API's for anyone interested. http://strongloop.com/strongblog/compare-express-restify-hap...

BradRuderman3• 12 years ago

Why are they loading in routes from an external source? Is that normal, I have never seen that before.

mjr5783• 12 years ago

We like the option of dynamically loading new routes, that point to new endpoints. We also have the ability to release new versions of our UI without redeploying (or restarting) our servers.

BradRuderman3• 12 years ago

Ok you add a new route but how do you reference what code should be executed when that route is hit?

dmak• 12 years ago

Could you elaborate in a bit on this? This is the first time I have heard of this, and I find it very interesting.

remon6• 12 years ago

I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it's not exactly an obvious choice.

joshuacc• 12 years ago

They went into this a bit on the NodeUp podcast: http://nodeup.com/seventyone

StevePerkins• 12 years ago

Are there some salient points that can be summarized here? That's an hour-and-twenty-minute podcast episode.

personZ• 12 years ago

Netflix seems to operate like a tech start-up that is trying to glue together a ragtag collections of often unsuitable solutions because of limited funding. It is a deeply perplexing company.

Similar is LinkedIn, as an aside -- despite being fairly formidable now, I regularly have entire feeds disappear, their caching is abhorrent, they can't markup text properly, and so on. It seems very amateur hour, yet they regularly publish "how it's done" documents that see wide applause despite often completely contradicting their prior missives.

general_failure6• 12 years ago

Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition.

The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.

yourad_io5• 12 years ago

What are the arguments against node.js in their use case?

Not looking to start any wars, but I was under the impression that if you know what you're doing* node.js is pretty awesome.

This particular bug had to do with a misunderstanding regarding the express API.

* for the most part: understand async and closures/memory leaks.

yourad_io5• 12 years ago

To give some arguments in favour:

- "Here we see our latencies drop down to 1 ms and remain there after we deployed our fix." <- Performance (from TFA)

- They can hire from a vastly larger pool of developers. Since they've got both budget and brand recognition, they should have no problems hiring high end JS guys & gals.

Terr_2• 12 years ago

I was doing some googling about this earlier, IIRC one of the big ones was "Good when your bottleneck is I/O, not good when your bottleneck is CPU."

tjholowaychuk3• 12 years ago

I share this thought, I'm not trolling, I really believe node is a bad solution for something like Netflix.

Node has its perks but for a money making machine that relies solely on being available and providing a good customer experience, not so much.

I can't imagine the ops nightmares at that size, one buggy code path and the entire cluster could be down. These are issues that drove me away from Node to Go, in my opinion Node has way too many issues to run in money-making scenarios.

nmjohn• 12 years ago

in my opinion Node has way too many issues to run in money-making scenarios.

You can say that about a lot of languages other than node. How many billion dollar companies have their software written in PHP - a language that many people would agree has far more glaring issues than node.

My point, is that I believe your comments miss the larger picture. There is far more involved with deciding which language to build a product with than "which language is best."

I don't disagree with you about Go being a great language - but for most companies, it is not even remotely practical to use. Hiring talented Go developers is hard because there are so few. Their current employees may or may not know Go, what do you do about that? Etc.

nthtran• 12 years ago

Could you expand more on the ops issues you have encountered in your systems?

bentcorner• 12 years ago

Interesting article. I have a lot of experience dealing with ETLs in WPA on the Windows side - it's an awesome tool that gives you similar insights. I haven't used it for looking at javascript stacks before though, so I don't know if it'll do that.

qodeninja• 12 years ago

wow. I love that Netflix us using Node and even more curious that they would use express.

elwell• 12 years ago

TIL, SVG's can display labels on element hover: http://cdn.nflximg.com/ffe/siteui/blog/yunong/200mins.svg

Nice, contained way to show data like this.

TazeTSchnitzel2• 12 years ago

SVGs can contain ECMAScript, video, canvases, animation, all sorts of things. They're the replacement for Flash that nobody seems to use.

wmil• 12 years ago

They've been around since 2001, but browser support only got good recently. Initially Adobe and Corel made browser plugins but Corel scaled back and Adobe bought Macromedia.

Also SVG GUI tools haven't been great. Inkscape is strongly print focussed, or at least was the last time I used it. There's no reason to set a page size on a web svg...

Someone needs to build an SVG equivalent of Flash Pro.

escape_goat• 12 years ago

Well, there was a good reason for that, for a long time: IE didn't even begin to support .svg until version 9, which means that it isn't a realistic option for deployment for anyone who needs to support pre-evergreen browsers.

I investigated them a year or two or so ago, because I found that AngularJS could work quite nicely within an .svg document, which opened up some exciting possibilities. My recollection is that at the time, there were some critical cross-browser problems with font rendering that made the topic very kludgy and complicated. I except that matters have improved since, but I do not know to what extent.

AdamTReineke• 12 years ago

There's a decent amount of JavaScript in there to enable it.

general_failure6• 12 years ago

A very good reason to go with express is TJ. He was the initial author of express and he is quite brilliant when it comes to code quality. Of course, TJ is no more part of the community but his legacy lives :-)

Pharohbot• 12 years ago

I wonder how Netflix would perform with using Dart with the DartVM. I reckon it would be faster than Node based on benchmarks I've seen. Chrome DartVM support is right around the corner ;)

codelucas• 12 years ago

This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array. Unfortunately, it was also inadvertently adding a static route handler with the same path each time it ran.

I don't understand the need of refreshing route handlers. Could someone explain they needed to do this, and also why from an external source?

Terr_2• 12 years ago

It sounds to me like some kind of configuration/IT-infrastructure-control issue. I wouldn't be surprised if it relates to some sort of "we want a way to toggle this live without an actual code-push" narrative.

untog• 12 years ago

I assume this is some kind of continuous delivery-style thing, where they can push up and amend endpoints without bringing down the entire system.

mjr5783• 12 years ago

We refresh periodically as we dynamically deploy new UI code, which can be accessed at new routes. (/home and /homeV2 for example) This allows us to not have to restart our servers or push out new server code just to serve a new UI at a different (or the same) route.

pcl• 12 years ago

Second, given a performance problem, observability is of the utmost importance

I couldn't agree with this more. Understanding where time is being spent and where pools etc. are being consumed is critical in these sorts of exercises.

coldcode• 12 years ago

I must admit I could enjoy just doing this type of analysis all day long. Yet I hate non computing puzzles.

hit8run• 12 years ago

I would have written my apis in golang and not nodejs. Go is way faster in my experience and it feels leaner to create something because creating a web service can be productively doneout of box. Node apps tend to depend on thousands of 3rd party dependencies which makes the whole thing feel fragile to me.

augustl2• 12 years ago

A surprising amount of path recognizers are O(n). Paths/routes are a great fit for radix trees, since there's typically repetitions, like /projects, /projects/1, and /projects/1/todos. The performance is O(log n).

I built one for Java: https://github.com/augustl/path-travel-agent

kyllo2• 12 years ago

How does your radix trie implementation handle variables in the URL paths, in a nutshell?

augustl2• 12 years ago

This is one of the reasons for why I call it a "bastardized radix tree" in the README :)

The routes are stored as "nodes". There's a root node. It has a hash map of child nodes, by name. It also has a list of "parameterized" nodes. When a node gets a path segment, it will first look in its hash map. If nothing is there, it'll call the parameterized nodes in sequence. Typically there's just one parameterized node.

For the following paths:

  /projects
  /projects/new
  /projects/special
  /projects/:project-id

The root node will have a single item in its hash map, "projects". No items in the parameterized node.

The node for "projects" will have to items in its hash map, "new" and "special". It will have a single item in it's parameterized node, for :project-id.

I updated the README just now with a slightly more detailed explanation :)

dmitrygr• 12 years ago

So the lesson is to actually know the code you deploy to prod? Is that not obvious?

kin• 12 years ago

Not everyone digs into the code behind their dependencies.

SixSigma• 12 years ago

Your webserver, your dns resolver, your database, your operating system, the compilers they used, how about the bios, or the northbridge

serve_yay• 12 years ago

Everyone should read the terms & agreements too.

talkingtab• 12 years ago

an unfortunate title. Ha ha "flames" ha ha "Node.js" but the article is really about express. Not so "ha ha"

snlacks• 12 years ago

They renamed it "in flame graphs."

For the audience, this was originally titled: "Node.JS in Flames"

wpietri• 12 years ago

From the article:

What did we learn from this harrowing experience? First, we need to fully understand our dependencies before putting them into production.

Is that the lesson to learn? That scares me, because a) it's impossible, and b) it lengthens the feedback loop, decreasing systemic ability to learn.

The lesson I'd learn from that would be something like "Roll new code out gradually and heavily monitor changes in the performance envelope."

Basically, I think the approach of trying to reduce mean time between failure is self-limiting, because failure is how you learn. I think the right way forward for software is to focus on reducing incident impact and mean time to recovery.

quaunaut• 12 years ago

I think the right way forward for software is to focus on reducing incident impact and mean time to recovery.

So in this case, guarantee you have a strong means of evaluating performance and maybe even include it by default just to be sure.

akkartik2• 12 years ago

Without over-training on this one incident, and without guidance on how to get from here to there (I'm still working on that):

1. Don't get suckered by interfaces, share code. If you create code for others to share ("libraries"), stop trying to hide its workings.

2. You don't have to learn how everything works before you do anything. But you should expect to learn about internals proportional to the time you spend on a subsystem. Current software is too "lumpy" -- it requires days or months of effort before yielding large rewards. The first hour of investigation should yield an hour's reward.

3. "Production" is not a real construct. There will always be things that break so gradually that you won't notice until they've gone through all your processes. Give up on up-front prevention, focus instead on practicing online forensics. And that starts with building up experience on your dependencies.

More elaboration: http://akkartik.name/post/libraries2

My attempt at a solution: http://akkartik.name/about

My motto: reward curiosity.

rwaldin• 12 years ago

I'm surprised nobody has mentioned that express has a built in mechanism for sublinear matching against the entire list of application routes. All you have to do is nest Routers (http://expressjs.com/4x/api.html#router) based on URL path steps and you will reduce the overall complexity of matching a particular route from O(n) to near O(log n).

pm90• 12 years ago

I love these kinds of investigations into problems in production. I mean, you really have to admire their determination in getting to the root of the problem.

In some ways, these engineers are not that different from academic researchers, in that they are devising experiments, verifying techniques, all in the pursuit of the question: why?

exratione• 12 years ago

The express router array is pretty easy to abuse, it's true. For example, as something you probably shouldn't ever do:

https://www.exratione.com/2013/03/nodejs-abusing-express-3-t...

I guess the Netflix situation is one of those that doesn't occur in most common usage; certainly dynamically updating the routes in live processes versus just redeploying the process containers hadn't occurred to me as a way to go.

sysk• 12 years ago

We also saw that the process’s heap size stayed fairly constant at around 1.2 Gb.
Something was adding the same Express.js provided static route handler 10 times an hour.

Why didn't it increase the heap size? Maybe it was too small to be noticeable?

ChuckMcM4• 12 years ago

The moneyquote:

"We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue."

This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varies all over the map. Consequently any decision to use such an API seems to require one first read and review the entire implementation of the API, otherwise you get the experience that NetFlix had. That is made worse by good APIs where you spend all that time reviewing them only to note they are well written, but each version which could have not so clued in people committing changes might need another review. So you can't just leave it there. And when you find the 'bad' ones, you can send a note to the project (which can respond anywhere from "great, thanks for the review!" to "if you don't like it why not send us a pull request with what you think is a better version.")

What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software and they focus on their product innovation.

There was an interesting, if unwitting, simulation of this going on inside Google when I left, where people could check-in changes to the code base that would have huge impacts across the company causing other projects to slow to a halt (in terms of their own goals) while they ported to the new way of doing things. In this future world changes, like the recently hotly debated systemd change, will incur costs while the users of the systems stop to re-implement in the new context, and there isn't anything to prevent them from paying this cost again and again. A particularly Machievellan proprietary source vendor might fund programmers to create disruptive changes to expressly inflict such costs on their non-customers.

I know, too tin hat, but it is what I see coming.

darkandbrooding• 12 years ago

Please forgive me if I'm misinterpreting you, but the lament you make about open source software seems to me to be more about distributed systems. It is an interesting observation, and now that you've pointed it out I can observe the pattern at past and present employers. But I have seen that pattern on internal software, written by the company for the company. This makes me think the problem is an architectural one.

I hesitate to comment about the relative velocities of open source vs proprietary software because I do not have enough experience with commercial, third party software. My sample size is too small, but I'm inclined to agree with you.

I don't disagree with your Machiavellian conspiracy, either, but I've worked in marketing and advertising so I know that some of the villains are on the payroll. Maybe there needs to be a third category? There's open source software, written by someone who has no particular relationship to you. There's commercial software, written by someone who has a positive economic relationship with you. And then there is ... corporate?... software, written by someone who might think they have a zero sum relationship with you.

rgawdzik• 12 years ago

You can't misuse closed source APIs? How would you know something is O(n) without seeing the source code?

snowwrestler• 12 years ago

I wondered the same thing too. I think the answer is here:

people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software

I guess the theory is that if you have a customer/vendor, contractual relationship, then you can task the closed-source vendor to describe, fix, or alter their API.

But my (albeit limited) experience is that the customer is not always right; sometimes the vendor just says "that's the way it is" and you're stuck either living with it or porting to a different company/technology. Conversely, it's often possible to hire a vendor or consultant to provide expertise and support for some aspect of an open-source stack. And because it's open-source, if you don't like your consultant you can go find another one without having to change your technology.

In short, I think the speed of innovation thing might have more to do with internal teams trying to manage too much diversity of technology, than with some inherent shortcoming of open source.

vorador• 12 years ago

What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software and they focus on their product innovation.

To contrast with what you said, I've worked at Microsoft, which is almost the company that invented NIH and we had the same problem. I think it's because, to paraphrase Alan Perlis, programming nowadays is less about building pyramids than fitting fluctuating myriads of simpler organisms into place.

ChuckMcM4• 12 years ago

That is an excellent point. That we're building primarily distributed software rather than point software has two effects, one that the number of failure modes are quite high, and two that the 2nd and 3rd order interactions are not easily deduced by inspection. I like the organisms picture. I think of them as 'atoms' in a molecule but organisms is much better, it includes their own idiosyncratic behavior into the mix.

dantiberian• 12 years ago

Consequently any decision to use such an API seems to require one first read and review the entire implementation of the API, otherwise you get the experience that NetFlix had.

There's no getting around this, you or someone that you trust (not necessarily at your company) needs to read and review both the API, and implementation details for open source software that you use. Open source software isn't a hardware store you can dip into to get the latest parts that you need for your project. Adding a dependency makes your code depend on other peoples code. Just like internal code should go through a code review, so should external dependencies. This also explains why for a lot of companies, it's easier to write their own thing rather than need to keep on top of other people's changes.

I happened to write about this yesterday which explains my thoughts further http://danielcompton.net/2014/11/19/dependencies.

dpweb• 12 years ago

Especially in high availability/load systems like Netflix IMO you need to reduce complexity and that means less modules that depend on 69 other modules, each of which depends on 69 modules etc.. What's going on as this sw proliferates is insane.

Then, you've got to be intimately familiar with every piece. There's no excuse when the source is readily available. I give these guys credit though they seem to take responsibility instead of just saying "express sucks". Some of their design choices seemed a little shaky.

nissimk• 12 years ago

Actually, open source allows something that is not possible with commercial software. You don't have to read all of the code and understand all of the implementation details, but you have that option if you need to. I think having the option is a great benefit as we can see from this example. When they started debugging they could reference the express source and figure out what was going on. If they were using some sort of commercial framework they would have had to either refer to the docs or call the help desk.

Just because you have the option to read the implementation details of every library and service you are using doesn't mean that you have to. You only have to learn enough about it to decide whether you think it is a good addition to your stack, and to use it to do whatever you are trying to get done. But open source gives you the ability to figure out why you're using it wrong, or how it is broken when that time comes.

If you are saying that open source is not documented well enough because the developers fall back on "check the source," that is a different argument where commercial software may be better, but this is not true for any of the more common open source software I've used.

Making a decision of which software to depend on in your application is something that is always difficult whether you have access to the source code of all of the choices or not. It's a decision that you make with limited information. You only have extensive knowledge of the tools that you already have experience with, so using alternatives is always a risk, but understanding and managing that risk is part of the developer's job.

And with regard to keeping up with changes, you always can remain with previous versions for some time to avoid the slowdown associated with shifting to new APIs.

ChuckMcM4• 12 years ago

I agree with you that the greatest strength of open source is that you can just go read the code and (optionally) fix it. Netflix did this and it got them past their crisis.

I think you missed my point though, which was that there isn't any sort of fitness function being applied to open source. In the closed source environment that it 'price' (caveat walled gardens). By paying for the software customers "vote with their wallet" on the things they like/want/use. What is more if two folks are putting out the same capability they are incentivised to compete on a vague sense of value. That creates a 'culling' function which operates independently of the software creators (the only way to sell more is to be "better").

Open source doesn't have that forcing function, there isn't really even a reputation function involved where bad FOSS would give folks a bad reputation that a user could pick up on (yes we probably all know one or more committers who are jerks but that isn't quite the same thing). And there is no barrier to entry so we get multiple variations on the same conceptual solution (I've lost count of the number of CMS's or blogging packages there are out there for example).

Finally there is the various bits of entanglement, you wrote "And with regard to keeping up with changes, you always can remain with previous versions for some time to avoid the slowdown associated with shifting to new APIs." but that has not been my experience. As you're evolving your product interdependencies in the APIs and packages you are using force them to upgrade as well. There is no island of stability in the FOSS world (well maybe OpenBSD servers or something but I have not seen them in the 'stack') Backports only go on so long and suddenly you're no longer getting updates from your source as Ubuntu showed when they dropped updates for 12.x LTS. All those systems still shell shockable or heart bleedable, or something else. Because to upgrade that part, needs to upgrade the next part, and so on and so on until you've upgraded everything. And you pay that price again and again and again and again.

Much of this is just different than the other model which is not necessarily a bad model. But from a systemic point of view it is hard to partition the work and that leads to inefficiencies. A single change in an API implementation which results in dozens of engineering groups re-implementing stuff, versus a migration model that allows people to move over gradually. The promise of open source is "no capital expense" but I worry the down side will become "much higher operational expense". And when the opex cost of open source is > than the capex + opex cost of closed software, then closed software wins. Of the folks who have tried to prevent that equation from shifting in favor of closed source, I only see Redhat as a success story.

akkartik2• 12 years ago

You're assuming that your closed source vendors are perfectly aligned with you. In practice they almost inevitably seem to cause capture (https://en.wikipedia.org/wiki/Regulatory_capture).

Open/closed is a red herring here. Projects slowing down as they succeed seems to be a universal phenomenon, from startups to civilizations. Specialization leads to capture. I think almost exclusively about how to fix this: http://akkartik.name/about (you've seen and liked this), http://www.ribbonfarm.com/2014/04/09/the-legibility-tradeoff

Disclosure: google employee

drawkbox• 12 years ago

I agree this is just part of the software iteration game. This has been Microsoft's modus operandi since they started. Microsoft still is with getting people to move to Azure/mobile, the cycling is just happening further up the stack.

Some of the iterations are good, some are just for control, mostly though it is to keep developers locked in and chasing rather than innovating. Software has to change and iterate though, but too slow or too fast can be harmful or the worthiness of iterations varies when platforms change.

On the topic of Node.js and Express though I think it is production solid and it is very fast (faster than php, ruby, python). I love it and I think here Netflix developers are at fault for not testing something a bit outside of the express default usage that is tested thoroughly. Their solution of changing to Restify so quick on a problem probably tells you how they got into this problem even if Restify is indeed better for this purpose. With any new change comes research/testing/possible problems open source or closed.

In the end this is why microframeworks, which both of the node frameworks are, do win out as they are easier to inspect and live on after the hype (monolithic frameworks not so much).

nostrademons10• 12 years ago

Yep, closed source doesn't solve the problem either. If you believe that just because you're paying money for someone to take responsibility for a problem, they will actually solve the problem in a way that's amenable to you...well, there are numerous closed-source software vendors looking to sell you something.

In practice, the way to avoid this is to keep the software as simple as possible. Try to adjust to your user's most pressing current needs, not every need they might conceivably have. Killing features and deleting code is as important as launching features and writing code; make sure that your incentive systems reward this. Very often, third-party code gets pulled in to scratch one particular itch; if it's no longer itching, rip the code out. If it is still itching and you've built significant parts of your system around it, you may want to think about replacing the innards with a home-grown system.

chx• 12 years ago

This is why I like choosing open source technologies with some sort of commercial support available. The best support I've ever gotten was from MySQL ab. (before Sun) -- the 10k we paid them for two years and three servers was affordable back then even for a small startup. I had a MySQL engineer (if memory serves he is now a MySQL community manager at Oracle) SSH into my server 34 minutes after a desperate call.

Disclaimer: I work for a company providing commercial support (scaling) for an open source project (Drupal).

scoates• 12 years ago

Morgan definitely knows what he's doing. (-;

runT1ME3• 12 years ago

What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software and they focus on their product innovation.

I think...your experiences at Google have altered your world view to the point where you don't see how thing are happening at other organizations. Google's monolithic codebase where everything builds against everything else may work(?) for them, but the alternative is disciplined module management.

You don't have to ever bump a version of working code if it's doing its job. Good open source projects should absolutely (publicly) test against performance regressions. New versions should be minor/incremental and source compatible.

I've never worked on a codebase the scale of Googles', but I fail to see how you can't mitigate your concerns, nor do I see commercial software the solution.

nostrademons10• 12 years ago

It's funny, my first job out of college was at a startup with an ex-Sun CTO, and he insisted on a single monolithic codebase. After he left I tried to organize an (aborted) project to modularize the codebase, since many of the other engineers ran into it as well.

Having worked at Google in the interim, and having a number of acquaintances at Microsoft (which uses the multiple-repository approach) I can see the pros and cons of both. The biggest benefit of the single codebase isn't technical, it's cultural. When you have a number of interdependent modules, then every change request and new feature has to go through that module's owner. If it's not their top priority (and it won't be), then getting your work done suddenly has a hard dependency on a team who is...generally pretty unresponsive, at least from your POV. The result is a lot of finger-pointing and political infighting, where every division thinks that every other division is a bunch of bozos.

The nice thing about Google's system (which, IIUIC, was Sun's as well, and is also Facebook's) is that when you have a hard dependency on another team and their priority list doesn't line up with yours, you can say "Well, can I make the change myself and you review it?" and the answer is usually yes. That means that people's default worldview is to assume busyness, not malice or stupidity, which makes the company as a whole function much better together. Yes, it creates a huge mess that someone will eventually have to clean up. But now you have a lot of options for how to clean it up, not a single point of failure: you can have the feature implementor do it, or the code owner, or it may get replaced entirely if the system is rewritten, or the feature may be unlaunched and no longer necessary, or someone may write an automated Clang or Refaster tool to fix a bunch of instances at once.

I'd compare it a lot to democratic capitalism: it's the worst system that exists, except for all the rest. When I was at Google, we all complained about how everybody else checked in changes that added complexity to our code. But if we couldn't do that, we'd all be complaining about how everybody else prevented us from getting our work done. I know which problem I'd rather have.

gcv• 12 years ago

This is precisely why — for some products in some industries — NIH is a reasonable strategy for writing good programs.

collyw2• 12 years ago

I would say this is more of a mature software versus new software rather that open versus closed source . Node isn't even at version 1 yet.

hardwaresofton• 12 years ago

Responses are already firing in: https://news.ycombinator.com/item?id=8632220

degobah• 12 years ago

tl;dr:

* Netflix had a bug in their code.

* But Express.js should throw an error when multiple route handlers are given identical paths.

* Also, Express.js should use a different data structure to store route handlers. EDIT: HN commentors disagree.

* node.js CPU Flame Graphs (http://www.brendangregg.com/blog/2014-09-17/node-flame-graph...) are awesome!

drinchev• 12 years ago

NodeJS Project has already a similar issue about recursive route matching.

https://github.com/strongloop/express/issues/2412

notastartup• 12 years ago

this is why you stick to tried and true methods folks. this is such a typical node.js fanboy mentality. "reinventing the wheels is justified because asynchronous". or "i want this trendy way to do things just because everyone else is jumping on the bandwagon".

Give me flask + uwsgi + nginx anyday.

CmonDev• 12 years ago

In the end of the day Node.js is just a Reactor Pattern implementation in a form of bunch of scripts.

_Marak_• 12 years ago

I read:

"This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array"

refresh our route handlers from an external source

This is not something that should be done in live process. If you are updating the state of the node, you should be creating a new node and killing the old one.

Aside from hitting a somewhat obvious behavior for messing with the state of express in running process, once you have introduced the idea of programmatically putting state into your running node you have seriously impeded the abiltity to create a stateless fault tolerant distributed system.

emeraldd• 12 years ago

When I concluded what they had to be doing and then read the actual confirmation of what they were doing I was somewhat shocked. Why on Earth would you want to programatically recreate the routes in an express app?!?!? It would be really interesting to see a write up on what/why they think this kind of behavior is needed in the first place ....

bcoates3• 12 years ago

It's not just the extra lookups -- static in express is deceptively dog-slow. For every request it processes, it stats every filename that might satisfy the URL. This results in an enormous amount of useless syscall/IO overhead. This bit me pretty hard on a high-throughput webservice endpoint with an unnoticed extra static middleware. I wound up catching it with the excellent NodeTime service.

Now that I look at it, there's a TOCTOU bug on the fstat/open callback, too: https://github.com/tj/send/blob/master/index.js#L570-L605

This should be doing open-then-fstat, not stat-then-open.

revelation• 12 years ago

Crazy talk. In 1ms, I can perspective transform a moderately big image. NodeJS cant iterate through a list.

We really need a 60 fps equivalent for web stuff. You have 16ms, thats it.

nostrademons10• 12 years ago

FWIW, a lot of the stuff the Chrome+Polymer team is working on explicitly has a 60fps goal and 16ms frame budgets. I remember in my last project at Google, I spent a lot of time working with the Chrome team to get various parts of websearch rendering under the 16ms budget. (I couldn't manage it given the amount of legacy code we had to work with, but various other people have continued the work since I left, so hopefully they've had more luck.)

Fishrock123• 12 years ago

I would like to mention that Netflix could have consulted the express maintainers (us) but didn't.

Source: myself - https://github.com/strongloop/express/pull/2237#issuecomment...

tjholowaychuk3• 12 years ago

Sounds like a documentation issue, or lack of a staging environment. I've written and maintained countless large Express applications and routing was never even remotely a bottleneck, thus the simple & flexible linear lookup. I believe we had an issue or two open for quite a while in case anyone wanted to report real use-cases that performed poorly.

Possibly worth mentioning, but there's really nothing stopping people from adding dtrace support to Express, it could easily be done with middleware. Switching frameworks seems a little heavy-handed for something that could have been a 20 minute npm module.

jaytaylor2• 12 years ago

I am upset that the title has been changed from "Node.js in Flames". Which is not only the real title of the article, but also a reasonable description of what they've been facing with Node.

#moderationfail

dang• 12 years ago

I can see why you would feel that way, but the title, while clever and (I'll take your word for it) fitting, was arguably misleading and unarguably baity. The HN guidelines call for changing such titles, so the moderators were just doing their job. There likely would have been more complaints about the title if we hadn't changed it.

jaytaylor2• 12 years ago

Understood. And thank you for following up, I really do appreciate it.

clebio• 12 years ago

I can’t imagine how we would have solved this problem without being able to sample Node.js stacks and visualize them with flame graphs.

This has me scratching my head. The diagrams are pretty, maybe, but I can't read the process calls from them (the words are truncated because the graphs are too narrow). And I can't see, visually, which calls are repeated. They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?).

At least, I _can_ imagine how you could characterize this problem without novel eye-candy. Use histograms. Count repeated calls to each method and sort descending. Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense. It sounds more like 'instrumentation', decorating the function calls to produce additional debugging information. Either way, once you have that, there are various common ways to isolate performance bottlenecks. Few of which probably require visual graphs.

There's also various lesser inefficiencies in the flame graphs: is it useful (non-obvious) that every call is a child of `node`, `node::Start`, `uv_run`, etc.? Vertical real-estate might be put to better use with a log-scale? Etcetera, etc.

donavanm• 12 years ago

The diagrams are pretty, maybe, but I can't read the process calls from them (the words are truncated because the graphs are too narrow).

Flame Graphs provide SVGs by default. You should be able to zoom if your broser supports it. The current version also supports "zooming" in to any frame in the stack, resetting that frame as the base of the display. Also WRT the base frames of 'node' et al its because Flame Graphs are a general use tool for stack visualization, it might be 'main' for a c program or the scheduler looking at a system.

They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?).

Colors by default have no meaning and the palette is configurable. The current lib can also assign colors by instruction count/ipc and width by call count, if you have access to that.

Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense.

It is sampling. Flame graphs re typically used with something like perf/dtrace/oprofile which dumps stacks at a couple hundred to thousand hertz. Actual call tracing is (typically) not feasible for large/prod stacks.

jasonkester• 12 years ago

You're looking at a screenshot. The actual diagram isn't static. Hover over it and it will expand each box with all the info you need to see what was run, what called it, etc.

MichaelGG• 12 years ago

Would someone explain what I'm missing about the flame graphs? Why are they indispensable here? In a normal profiler, you'd just expand the hot path and see what had the most samples. Apart from making recursion very explicit, what special aspect do flame graphs expose?

Node.js in Flame Graphs

Related Stories