LinkedIn Mobile Moved from Rails to Node: 27 Servers Cut and Up to 20x Faster

http://highscalability.com/blog/2012/10/4/linkedin-moved-from-rails-to-node-27-servers-cut-and-up-to-2.html

by turar • 14 years ago

204 136 14 years ago

justinjlynn2• 14 years ago

It sounds like they went through a major rewrite of their backend and ended up architecting things to be much more performant than their previous system. I'm curious to find out what parts of the system they think contributed most to the performance increase. While this is interesting it is by no means an apples to apples comparison of Node and Rails as the headline suggests.

some1else• 14 years ago

The aggressive caching approach taken by Basecamp Next is another example of such an optimization: http://37signals.com/svn/posts/3112-how-basecamp-next-got-to...

aantix• 14 years ago

DHH shows off some of the New Basecamp caching logic in this video :

http://www.youtube.com/watch?v=FkLVl3gpJP4#t=33m30s

gnufied4• 14 years ago

From original article[1] (Not the highscalability link spam):

>This led them to a model where the application is essentially piping all of its data through a single connection that is held open for as long as it is needed.

So it sounds like from using a request/response driven architecture, they adopted streaming architecture. Also, they moved away from Rails and adopted an Event driven approach.

It is well known fact that, it is almost impossible to stream stuff with Rails currently and hence adopting a event driven approach made sense. I can see, how just these two factors alone will contribute hugely to performance.

[1] http://arstechnica.com/information-technology/2012/10/a-behi...

rhizome3• 14 years ago

Sure, but is it unreasonable to wonder why they need such an architecture? LinkedIn is basically a CRUD app, and while they have a wall now and yadda yadda, I wonder how much of this rearch was really necessary, over simple refactorings and sysadmin attention to the nuts and bolts.

justinjlynn2• 14 years ago

Indeed, shifting to a streaming architecture can vastly reduce the amount of churn your application sees. This makes great sense if it fits the use case of your product. However, one need not throw out a huge existing code/knowledge base in order to accomplish this shift. I think that it's perfectly acceptable to shift to something like EventMachine + some HTTP layers to implement a streaming server using a good chunk of your existing codebase and developer experience. Node.js is the new hotness and so is the prospect of rewriting your application cleaner, faster and better... but often, the less you have to rewrite or learn, the more likely it is that your project will be successful.

SoftwareMaven4• 14 years ago

If you are thinking about using node.js for this reason[1] on most sites, you are optimizing poorly. LinkedIn didn't worry about this until after they were a public company.

If Python/Djando or Ruby/Rails can get your app out the door and into customer hands faster, it is almost always the right thing to use.

1. There are certainly other, very valid, technical reasons for choosing node.js over other technologies early. But let those reasons be about the problems you are solving today, not the ones you might need to worry about when you have 50 servers to deal with.

Smudge• 14 years ago

I'm sure LinkedIn could have cut servers without switching to node. Take your first, crappy implementation and rewrite it in the same language and you'll probably still see at least 10x improvement, if not 20.

asianexpress• 14 years ago

My thoughts exactly -- going from 30 to 3 servers is no joke and it couldn't just be because of a move to node.js

nicholassmith2• 14 years ago

This times a thousand. When you first write the software you're basing it on expectations, no matter how well you plan, but when you rewrite it you're coming at it with real world knowledge of the pain points so of course it's going to be better in terms of performance.

ericcholis• 14 years ago

I'm with you on this one. I'm currently in the process of re-platforming and I'm noticing considerable gains just from revising the way certain process are done. You find a lot of "wtf was I thinking".

pdelgallego• 14 years ago

>> Take your first, crappy implementation and rewrite it in the same language and you'll probably still see at least 10x improvement.

You will probably will get the same mess. There is plenty of literature about that. I.e: A complete rewrite is what killed Netscape

dkrich• 14 years ago

I disagree. When you change state management to client side, you are making a fundamental architecture shift that is significant enough to remove a lot of server-side overhead. What makes you think refactoring code is going to give you a 10x improvement in efficiency? If your code is that bad, you should get rid of the developers along with the code.

fleitz• 14 years ago

Yeah as the article notes the old version used HTML and the new version uses binary blobs.

This is hardly surprising.

benologist4• 14 years ago

Node has some pretty unique benefits, I save a good $1000/month from switching from .NET on dedicateds to NodeJS on a PaaS (Heroku), and that was after 2 - 3 years of writing and rewriting and optimizing the .NET stuff.

Biggest improvements came from persistant connections to redis/mongodb and polling for updated information independently of requests so there was no cached-or-fetch shenanigans at all in some areas.

throwa• 14 years ago

This article actually gives details of their first implementation which suggest that they could cut servers without switching to node :

http://ikaisays.com/2012/10/04/clearing-up-some-things-about...

lumberjack• 14 years ago

To be fair, Node.js isn't exactly harder to work with than Django or Rails. I myself had to choose between these about a year ago and despite the asynchronous paradigm I still found it relatively easy to start using in comparison to Rails and Django.

astrodust4• 14 years ago

Node.js is extremely hard to work with compared to Rails and moderately hard to work with compared to Django.

The Node.js ecosystem has a lot of potential, but the variety of off-the-shelf add-ons is severely limited compared to either of those more mature frameworks.

If you think Node.js is easy, you've never really experimented enough to understand what makes Rails so effortless. It's a lot easier to produce a production-ready application with Rails than it is with Node.js as it is today.

In four years, as Rails starts to add less critical features, Node.js may well have caught up.

jonpaul• 14 years ago

Why not write your code using Node.js because you actually enjoy using JavaScript? Or how about the convenience of writing in one language in your stack? Or how about the simplicity provided by not having to worry about threading per process?

If you already know Python or Ruby, then ya, using Node.js would be silly. But if you have to pick one because you're equally familiar with all of the languages, then Node.js isn't a bad choice at all.

SoftwareMaven4• 14 years ago

You'll note I explicitly said there are reasons to choose it. And if you choose it for those reasons, great. Just don't choose it because it will solve your Mazzerati problem. Choose it for your problems today.

andersbreivik• 14 years ago

" because you actually enjoy using JavaScript? "

You sick bastard!

todd3834• 14 years ago

After reading the article, the fact that they cut servers was just one of the advantages they got from using node.js

kin3• 14 years ago

What I saw as a plus was that they combined front/back end teams into a single unit.

jeremyjh4• 14 years ago

Am I the only person who doesn't believe this story? Because really, the whole reason we have front-end/back-end specialists is because front-end javascript ninjas can't learn RUBY??!?

markmm8• 14 years ago

I would spend a little extra time upfront and save many many expensive rewrites latter on. Use C from the start, you won't need to rewrite just scale with hardware.

This argument that using Ruby (or any language that makes programming easy but runs like treacle) to get something out the door a few months earlier is bullshit, I would rather release something a few months later that I didn't have to totally rewrite down the line 10 times.

ollysb• 14 years ago

This would only make sense if you're working for an established company that can burn a few extra months early on. For a startup the runway might be too short. In this case you have no idea if your product is even going to exist a year later so it seems a bit premature to be worrying about the possibility of rewrites.

jonny_eh• 14 years ago

But the odds of the thing you release actually getting used are low. So make it fast, and if it's a hit, then deal with that.

andybak3• 14 years ago

I look forward to seeing C as the go-to web-development language.

Really? C for a problem that is largely string manipulation and database access?

SoftwareMaven4• 14 years ago

Said with true engineering "vision".

The reason why "getting it out the door" is so important is because nobody behind the door generally has any idea about what people will really pay for. By making it fast (read: cheap) to make, if you find that you don't get it right, you can try again, and again, and again.

Now, you can argue that "they shouldn't be releasing without knowing they will be successful", but if they have the ability to see the future, they should just take that VC money and put it in the market. People like Steve Blank and Eric Ries have a lot of evidence how running most startups (software startups particularly) using a "build it and they will come" attitude is ripe with failure.

Now, if you are just building for fun, that's different. I knew a guy building a Dropbox clone in C. More power to him (though I wouldn't have touched it, because I think it is too hard to get security right all the time in a language like C).

yxhuvud• 14 years ago

What makes you think you know the problem space enough to make a C implementation that you won't have to rewrite 10 times? Writing in C certainly doesn't stop any rewrites at least where I work.

burke• 14 years ago

I really need to quit reading HN comments. This entire thread is enraging. Both sides. Especially yours.

dpcx• 14 years ago

I personally find it hard to believe that all of LinkedIn was only running on 30 servers, and is now running on only 3.

EDIT: Mobile only. Maybe the title should be updated to reflect that.

masklinn• 14 years ago

> I personally find it hard to believe that all of LinkedIn was only running on 30 servers, and is now running on only 3.

I can believe it, if their caching strategy was garbage before and part of a massive rewrite was rethinking that. If you fail to use your caches, you'll pay for it.

I mean it's not like LinkedIn is about split-second changes, you could probably statically generate most of it and serve it with a single nginx instance.

ikailan4• 14 years ago

Actually, we aggressively cached, which led to weird things happening. I posted a follow up about this explaining more context - I hope it gets voted up.

wardb• 14 years ago

Reading this like: "Hey, our previous backend was a total turd, technically speaking." It might be trivial to speed up your own crappy first implementation 20x with some extra TLC.

rhizome3• 14 years ago

Like a lot of contemporary online companies, they might be skimping on sysadmins.

cmccabe• 14 years ago

yeah, but it's easier politically to rewrite it in a newer language. If you rewrite it in the same language, then you have to explain your rewrite as cleaning up someone's braindead mistakes-- someone who might still work there.

If you rewrite it in a different language, then you get to blame the old language and framework for all the problems. No harm, no foul-- and you get to put the new shiny thing on your resume.

benbjohnson• 14 years ago

Can someone update the title to read: "LinkedIn Moved from Server to Client: 27 Servers Cut and Up to 20x Faster"?

hnriot• 14 years ago

But that would be wrong, the server legitiamtely moved from rails to node. Just because node is JavaScript doesn't mean that it automatically runs on the client, JavaScript is a perfectly viable server language too.

The title could be improved by reflecting that this is for LinkedIn mobile, but the rails->node is correct.

ericf• 14 years ago

Not wrong, a lot of their performance gains were a result of caching their templates client side and having their servers only return data. That is a much smaller load on the server and easily could account for the reduction in servers. The title makes it sound as if they reduced their servers and increased their performance simply by migrating from ruby to node.

skyebook• 14 years ago

It's a nice stat to see but I think this sort of comparison with "we moved our infrastructure of undisclosed age and unknown bloat to this new infrastructure built for the current problem domain" doesn't really do much for the ongoing conversation.

The article is touted as praise for a stack but my gut says that its really a smart restructuring of how they serve mobile. Either way, good on them for the efficiency boost.

gnufied4• 14 years ago

The title is misleading. From original article[1]:

> They found that Node.js, which is based on Google’s V8 JavaScript engine, offered substantially better performance and lower memory overhead than the other options being considered. Prasad said that Node.js “blew away” the performance of the alternatives, running as much as 20 times faster in some scenarios.

So according to original article, Node.js did not perform 20 times better compared to existing Rails based backend. According to Prasad, it performed 20 times better than alternatives of Node such as - Eventmachine & Python Twisted (they did evaluate both of them).

Now I am having hard time believing node.js can outperform Eventmachine or Twisted by 20 times. Most benchmarks I have seen and done tell me, node is marginally ahead. I would obviously like to see, what they benchmarked and how?

1: http://arstechnica.com/information-technology/2012/10/a-behi...

pepve• 14 years ago

I think they created a specific benchmark for their use case. Maybe they even implemented part of their requirements on both stacks.

megaman821• 14 years ago

They should really release those benchmarks. From what I have seen Python/Gevent performance is on par with Node.js and I doubt Ruby/Eventmachine is that far behind either.

If they found a 20x performance difference there is probably a bug in one of those competing libraries.

antonpug2• 14 years ago

I hate Rails. Node is the way to go. Node > Python > Rails

evilduck• 14 years ago

Library > Language > Framework?

antonpug2• 14 years ago

What I mean to say...in terms of modern web development, Javascript > Python > Ruby and Node > Django > Rails.

markmm8• 14 years ago

1 server running Nginx with modules + Postgresql == 100 servers running Ruby/Node/Some scripting shit slow language talking to some NoSql crap.

j95tin• 14 years ago

Having an opinion is fine, but you need to support it with an explanation and/or data.

SanjayUttam• 14 years ago

"Programmers could leverage their JavaScript skills."

I'm confused by that being an advantage - I guess that's better than using some language you don't know at all, but it's still a bit of a different approach to JS, no?

fomojola• 14 years ago

Ah, odd math point, but was I the only one who noticed this sentence:

"focus on simplicity, ease of use, and reliability; using a room metaphor; 30% native, 80% HTML; embedded lightweight HTTP server; "

pragmatic2• 14 years ago

It's mangled from the original article.

jemfinch• 14 years ago

When you make a software change that allows you to reduce your server pool from 30 to 3, you don't say, "27 servers cut". You say "Servers reduced by 90%." I have no idea what "27 servers" actually means: you could have been using a thousand servers for all I know.

scott_meade• 14 years ago

"simplicity is at the heart of LinkedIn’s mobile vision." Then the article goes on to describe complexities of the implementation. Maybe my old and cranky mind has lost touch with what people mean by "simple".

derwiki• 14 years ago

Without citing versions, it's hard to extract anything useful from this article. Rails 2.1 on ree-1.8.7 performs very differently than Rails 3.2 on 1.9.3-p194.

pragmatic2• 14 years ago

The original article:

http://arstechnica.com/information-technology/2012/10/a-behi...

And this is really intersting:

Finally LinkedIn tested, and ultimately chose to adopt, something surprising. The company embedded an extremely lightweight HTTP server in the application itself. The HTTP server exposes native functionality through a very simple REST API that can be consumed in the embedded HTML controls through standard JavaScript HTTP requests. Among other things, this server also provides access to the user’s contacts, calendar, and other underlying platform functionality.

rorrr• 14 years ago

Ruby is one of the slowest languages you can think of, while Javascript on V8 is only 2.3X slower than C++ (median).

http://shootout.alioth.debian.org/u32/which-programs-are-fas...

What I'm really surprised is that they got such a huge gain. Most projects I've worked on are DB or I/O bound. Maybe they store everything in RAM.

jamesaguilar• 14 years ago

They probably fixed query design issues at the same time as they moved implementations. It strikes me as hilariously unlikely that LinkedIn was CPU-bound.

ikailan4• 14 years ago

It was a huge gain because the original Rails servers were running on single threaded Mongrel and blocking on cross data center IO.

trekkin• 14 years ago

It's almost impossible to reach 1000 req/sec in Ruby/Rails. It's relatively easy to reach 50000 req/sec in C++. Assuming V8 is about 3x slower than C++, it is not surprising that a move Ruby/Rails->node.js gets 10x throughput improvement.

jrockway2• 14 years ago

Is 27 servers a lot?

elpee• 14 years ago

read all the comments, not a single mention about php, wtf world?

ikailan4• 14 years ago

I was on the team at LinkedIn when we first wrote the thing on Ruby on Rails. Here's my writeup containing some more context:

http://ikaisays.com/2012/10/04/clearing-up-some-things-about...

While I'll freely admit v8 is much faster than MRI Ruby, the efficiency gains are likely more related to 1) the rewrite factor 2) moving to non-blocking 3) the fact that the original server ... um, needed love

shn• 14 years ago

What this article talks about is more than a year old. When I was looking for a tool to start my project I was at the same distance to Python, Ruby and NodeJS and their ecosystem. So when I read about it a year ago I leaned towards NodeJS. So knowing the experience of others always help to certain people at certain point of their yet to unfold story but not to everybody all the time.

I am not unhappy with my choice but I do not have enough data to compare with other tools. I do not think a lot of people have either. Once you start with a tool you tend to keep it since you invested a lot of time learning it as well as developing something with it. I think few can afford switching tools (e.g. FB switched from HTML5 to native recently for their mobile interface).

rapind• 14 years ago

I looked into Node.js, Sinatra, and Go to handle API traffic for a mobile app a few months ago and did a lot of benchmarking. What I found during my tests, was that Go > (Node.js = Sinatra).

If I had wanted to add Rails to this comparison I would have compared apples to apples and used Metal instead of including the entire stack.

lsh123• 14 years ago

The performance improvements have probably NOTHING to do with node.js but with the re-architecture goals set by the team:

"For our inevitable rearchitecting and rewrite, we want to cache content aggressively, store templates client-side (with the ability to invalidate and update them from the server) and keep all state purely client side."

Better understanding of the problem and experience running the system were probably key for building the new high-performance architecture. Obviously, old one lacked these big advantages.

dschiptsov• 14 years ago

Replacing inefficient and bloated Rails, with custom coded framework, and byte-code interpreted Ruby for native code generating V8, losing in readability in an order of magnitude? Well, nothing to see here.

chmike• 14 years ago

What would they gain if they moved from node to nginx and a c++ with asio backend (e.g. http://cgi.sourceforge.net/) ?

VeejayRampay• 14 years ago

Again with that little war of frameworks? Man there's a lot of money and pride behind all this, dear god.

lquist• 14 years ago

Hmm...

Does going from 30 servers to 3 actually make an impact to a $13B company?

LinkedIn Mobile Moved from Rails to Node: 27 Servers Cut and Up to 20x Faster

Related Stories