GitHub's Unicorn Setup

http://github.com/blog/517-unicorn

by mqt • 17 years ago

162 61 17 years ago

github.com

boundlessdreamz2• 17 years ago

How does passenger handle restarts? Does it also allow a zero downtime restart ?

joevandyk7• 17 years ago

You run "touch #{RAILS_ROOT}/tmp/restart.txt".

That will restart the rails processes. No connections are dropped, and I've not seen any downtime.

latortuga• 17 years ago

To clarify, my understanding is that while rails is restarting, Passenger will queue all the requests that come in and begin processing them as soon as rails is ready. But yeah, zero downtime, it's pretty awesome.

dschobel• 17 years ago

When the Unicorn master starts, it loads our app into memory. As soon as it’s ready to serve requests it forks 16 workers. Those workers then select() on the socket, only serving requests they’re capable of handling. In this way the kernel handles the load balancing for us.

Wasn't there a heated debate here just the other day about the prefork model?

Guess it's at least back en vogue @github.

seiji• 17 years ago

It's connection pooling, not a fork-per-accept web server. Each worker is select/epoll/kqueue'ing for individual requests.

aaronblohowiak• 17 years ago

Right, but it is being handled by processes and not threads, which the argument was about the other day.

tptacek2• 17 years ago

Has any mainstream web server ever been pure demand-forked? Apache has been connection pooled since the '90s; the second edition of Unix Network Programming used it as a case study.

davidw5• 17 years ago

Except that Apache (and Passenger) don't just start up N worker processes - they can dynamically allocate them as the need arises. Perhaps this is less important for a site like github, which can plan exactly how many to use, but if you have more than one site on the same server, it's really convenient, and, IMO, makes Passenger the best choice for deploying Rails for most people.

So it appears as if this setup replicates a small portion of what Apache does without all the bells and whistles.

cakeface• 17 years ago

Its great to see companies sharing what they've learned from experience about system architecture. So many times this sort of stuff is very difficult to plan out and the only real way to get it right is through experimentation. Having first hand descriptions like this is a great resource if you are setting something up the first time.

I like that they posted their unicorn config file too!

defunkt3• 17 years ago

Also it doubles as documentation for the members of our team who aren't familiar with this part of the system :)

blasdel2• 17 years ago

I don't understand why people insist on architectures where otherwise-independent processes share a single socket.

You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

wmf• 17 years ago

Replacing N ports with one simplifies configuration.

I never understood the complex HAProxy in front of Apache in front of Nginx in front of Mongrel type setups that seem to be popular in the Rails world. Why not just use Unicorn? What value is GitHub getting from having Nginx in front?

defunkt3• 17 years ago

Unicorn is not for slow clients or static assets. That's what nginx is for. See http://unicorn.bogomips.org/PHILOSOPHY.html for info on Unicorn and slow clients.

nginx also has features like ESI, serving from memcached, and rate limiting which Unicorn does not (and doesn't need).

jasonwatkinspdx2• 17 years ago

Because Ruby 1.8 threading sucks you pay a large memory price (ie a process) for each concurrent request in flight. A fronting proxy allows your backends to write out the response as fast as possible and move on to another request while the proxy spoon feeds the response to slow clients.

Also, nginx is going to be more efficient for serving static files, though most larger apps will have broken such requests out to a separate set of domains likely serviced by a cdn.

defunkt3• 17 years ago

If each Unicorn worker listened on a different port, we'd have to use one of nginx's load balancing strategies (unless I'm misreading your comment).

We have not had success with them in the past which is why we employed HAProxy with mongrel.

blasdel2• 17 years ago

You read me correctly, I don't see why load balancing should be taking place down at the app server level.

Aren't you going to have multiple machines each with their own blessing of unicorns? You're still going to have to use some kind of load balancer in front of independent sockets.

jasonwatkinspdx2• 17 years ago

Sharing queues wins. It's as simple as that.

It's a general result in queue theory that you want one global queue as early as possible in the system. This is because requests don't take exactly the average service time to clear backends: it's a distribution. The more workers that can pull from a queue, the less the worst case service times affect the average service time.

One queue per worker with no global queue is the worst configuration and should be avoided if at all possible. Anyone who's run large reverse proxy installs knows this pain well.

The ideal system would be for the balancer machine(s) to hold the requests, and for backends to pull them in a sort of ping/pong fashion. I gather fuzed runs in a pattern like this, though I've not used it.

Telcom folks have analyzed this stuff in detail for the better part of a century. There's a lot of theory out there and it's surprisingly practical and applicable to real world web applications.

atambo• 17 years ago

Is there any benefit in using unicorn over passenger?

fizx2• 17 years ago

Unicorn doesn't make your site very slow for the 5 seconds-2 minutes (YMMV) after you deploy.

chadr2• 17 years ago

It seems to me like the Passenger guys could easily add an option so that on a "touch tmp/restart.txt" a set of new worker processes is started before the old ones are killed off. I imagine this would make this slowness a thing of the past. For the record, my apps experiencing this momentary queueing and slowness on restart (5 seconds max).

Cornify• 17 years ago

The Grand Unicorn is proud!

GitHub's Unicorn Setup

Related Stories