Reining in the thundering herd: Getting to 80% CPU utilization with Django

https://blog.clubhouse.com/reining-in-the-thundering-herd-with-django-and-gunicorn/

by domino • 5 years ago

160 135 5 years ago

blog.clubhouse.com

vvatsa• 5 years ago

ya, I pretty much agree with 3 suggestions at the end:

* use uWSGI (read the docs, so many options...)

* use HAProxy, so very very good

* scale python apps by using processes.

catillac• 5 years ago

Famous last words, but I get the sense that the need to handle this sort of load on Clubhouse is plateauing and will decline from here. The app seems to have shed all the people that drew other people initially and lost its small, intimate feel and has turned into either crowded rooms where no one can say anything, or hyper specific rooms where no one has anything to say.

Good article though! I’ve dealt with these exact issues and they can be very frustrating.

hnarayanan• 5 years ago

This

latchkey4• 5 years ago

If it is just a backend, why not port it over to one of the myriad of cloud autoscaling solutions that are out there?

The opportunity cost of spending time figuring out why only 29 workers are receiving requests over adding new features that generate more revenue, seems like a quick decision.

Personally, I just start off with that now in the first place, the development load isn't any greater and the solutions that are out there are quite good.

_tom_• 5 years ago

They are on a back-end that does auto-scaling. They stated that they had problems when scaling up past 1000 nodes.

Now, maybe they could have fixed that issue instead, but going from 29 to 58 workers is easy, it's not the same going to 29,000 to 58,000. And 1000 hosts vs 500 is a non-trivial cost.

PaywallBuster4• 5 years ago

containers would've solved it

one process per container, easy peasy

motoboi2• 5 years ago

you now containers are just processes, right?

This is what they did, but because they didn't need to schedule other jobs on the same machine, kubernetes or even docker would be overkill.

In this case, simple VM orchestration seems like a fine solution.

Spivak2• 5 years ago

This doesn’t work so easily with architectures with process pools for workers. So now your app server needs to speak docker (or whatever control plane) to spawn new workers and deal with more complicated IPC. Also the startup time is brutal.

One process per container and multiprocessing is a huge lift most of the time. I’ve done it but it can be a mess because you don’t really have as much a handle on containers than subprocesses because you can only poke them at a distance through the control plane.

lddemi3• 5 years ago

Author here. We do and did use autoscaling heavily but at a certain scale we just ran out of headroom on the smaller instance types we were using. Jumping to a much larger instance types meant that we will likely never run into those headroom issues again, plus solves other problems like faster spin up, better sidecar connection pooling and allows for a much higher hit rate on per instance caching.

TekMol6• 5 years ago

Did you consider switching form CPython to Pypy?

latchkey4• 5 years ago

You were autoscaling a single threaded process. You had 1000 connections coming in and scaling 1000 workers for those connections. Everything was filtered through gunicorn and nginx, which just adds additional latencies and complexity, for no real benefit.

What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it. I'm talking about not running your own infrastructure, at all. Let AWS and Google be your devops so that you can focus on building features.

TekMol6• 5 years ago

Performance is the only thing that is holding me back to consider Python for bigger web applications.

Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake.

Pypy could be a solution as it seems to be about 6x faster on average.

Is anybody here using Pypy for Django?

Did Clubhouse document somewhere if they are using CPython or Pypy?

domino• 5 years ago

Clubhouse is using CPython

TekMol6• 5 years ago

Interesting. Is there a reason for this?

fnord773• 5 years ago

The top 5 web app programming languages by market share are PHP, Java, JS, Lua and Ruby.

TekMol6• 5 years ago

...said a stranger on the internet without any sources to back up this claim.

fnord773• 5 years ago

lol at the downvotes. sorry your favorite language isn't on the list.

I specifically said "market share", not "best" or "favorite".

https://www.wappalyzer.com/technologies/programming-language...

IshKebab4• 5 years ago

Typescript is a nicer language than Python in many ways and it doesn't suffer from Python's crippling performance issues or dubious static typing situation. Plus you can run it in a browser so there's only one language to learn.

TekMol6• 5 years ago

You cannot run TS in a browser.

You can compile it to JS or to Webassembly. But you can do that with every language.

franga2000• 5 years ago

Typescript would be nice if it weren't essentially just a bunch of macros for JavaScript. As it is now, as soon as you want to run it, you lose all the benefits of it (including many performance optimizations that could be made in a statically typed runtime) and of course, all the usual footguns of vanilla JS still apply. It's a great development tool though, I'll give you that.

mixmastamyk• 5 years ago

Backend controller performance is rarely a bottleneck, and if raw-compute still is there are a number of ways to speed it up, such as cython and/or work queues.

zinodaur• 5 years ago

I feel like people let go of perf too easily.

When using something like Golang, I have apps doing normal CRUD-ish queries at 10k QPS, on 32c/64g machines. For most web apps, 10k QPS is much more than they will ever see, and the fact that it is all done in a single process means you could do really cool things with in-memory datastructures.

Instead, every single web app is written as a distributed system, when almost none of them need to be, if they were written on a platform that didn't eat all of their resources.

jusssi• 5 years ago

People have been taught to be irrationally afraid of in-process concurrency (including async). Not too long ago the standard approach for concurrency was "it's hard, don't do it".

I've been told off in code review for using Python's concurrent.futures.ThreadPoolExecutor to run some http requests (making the code finish N times faster, in a context where latency mattered) "because it's hard to reason about".

midrus• 5 years ago

I could rephrase you comment as why would anyone use Go when I could just use assembler or C and keep all into a single node.

People don't use python because they want performance. People use python because of productivity, frameworks, libraries, documentation, resources and ecosystems. Most projects don't even need 10k qps, but instead most projects do need an ORM, a migrations system, authentication, sessions, etc. Python has bottle tested tools and frameworks for this.

sdze5• 5 years ago

use PHP ;)

nsizx• 5 years ago

So much this. Practically any other option is better than Python for web development if you're looking for performance.

void_mint5• 5 years ago

By this logic, why not Java, C++, Rust, Go, C#?

They’re all web-capable and blow the doors off PHP, Python, etc.

sdze5• 5 years ago

It blows my mind how quickly PHP7.4 processes even shitty code.

waprin• 5 years ago

Yet YouTube, Instagram, Pinterest, Reddit, Robinhood, DoorDash, and Lyft backend were originally primarily written in Python. What’s funny is that nobody can really deny Python is slow yet somehow the biggest websites in the world were written in it. More proof that Worse Is Better?

tbrock• 5 years ago

Aside: AWS only allows registering 1000 targets in a target group… i wonder if thats the limit they hit. If so, its documented.

dilyevsky4• 5 years ago

Kinda funny they decided paying a ton of money to aws was ok but paying for nginx plus was not

Spivak2• 5 years ago

I kinda get that honestly. It’s why I’ll spend $20 without even thinking for take out but not spend $2 for an app. It’s because the cost off the software is way way more than the money. It’s a commitment to actually use it and integrate it, deal with their sales team, talk to purchasing, handle licensing, and introducing friction to replacing it or using tools that don’t integrate well because “well we already pay for it.” Licensing also complicates deployments substantially when you’re doing lots of autoscaling.

And on top of that Nginx Plus is also expensive as hell.

dilyevsky4• 5 years ago

Don’t you have to integrate cloud? This whole post is about having to put a bunch of workaround bc the cloud can’t scale apparently

ClumsyPilot• 5 years ago

"It’s why I’ll spend $20 without even thinking for take out but not spend $2 for an app."

I pay for apps, its not a healthy attotude

rowanG077• 5 years ago

The buy in into AWS is much, much larger then using a piece of software though.

spullara• 5 years ago

The difference people see, as far as I can tell, is that AWS is charging you cost+ and pure software companies need to charge for value or die.

dilyevsky4• 5 years ago

Maybe for barebones compute they’re cost+ but I don’t think that’s really true for other services. For example traffic should cost effectively zero to them but they charge a huge premium. Some other managed services also appear to use value based pricing

andrenotgiant• 5 years ago

ClubHouse runs on AWS?

dilyevsky4• 5 years ago

Hm actually might be google based on what their traffic is going to (i only looked just now). Ok now it makes more sense why support wasn’t able to figure this out =)

stingraycharles• 5 years ago

Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled.

And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the door immediately.

Ozzie_osman• 5 years ago

Yea you are right. It could be a service being down and requests piling up, or a cache key expiring and many processes trying to regenerate the value at the same time, etc.

I think the article just used this phrase to describe something else. (Great article otherwise).

taylorhughes2• 5 years ago

Phrase borrowed from excellent uWSGI docs https://uwsgi-docs.readthedocs.io/en/latest/articles/Seriali...

fanf2• 5 years ago

There is an explanation of this kind of thundering herd about 3/4 down this article https://httpd.apache.org/docs/trunk/misc/perf-scaling.html

The short version is that when you have multiple processes waiting on listening sockets and a connection arrives, they all get woken up and scheduled to run, but only one will pick up the connection, and the rest have to go back to sleep. These futile wakeups can be a huge waste of CPU, so on systems without accept() scalability fixes, or with more tricky server configurations, the web server puts a lock around accept() to ensure only one process is woken up at a time.

The term (and the fix) dates back to the performance improvement work on Apache 1.3 in the mid-1990s.

thaumasiotes2• 5 years ago

but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled.

That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and your inbox is full enough for three inboxes. Does that fact, in itself, mean that you decide you're done for the day? No, it just means you have a much longer queue to work through than usual.

The thundering herd problem refers to what happens when (1) a bunch of agents come to you for something while you're busy; (2) you tell them all "I'm busy, go away and come back later"; and (3) the come-back-later time you give to each of them is identical, so they all come back simultaneously.

And that's exactly what's happening here, except that instead of giving each worker thread a come-back-later time when it asks for work, you're receiving work, sending out individual messages to every worker saying "hey, I'm not busy anymore, come back RIGHT NOW and get some more work", and then rejecting all but one of the thundering herd that shows up. The reason the Gunicorn docs and the uWSGI docs both refer to this as a "thundering herd" problem is that it's a near-perfect match for the problem prototype. The only difference is that, instead of giving out identical come-back-later times to worker threads as they ask you for work, you tell them to wait for a notification that includes a come-back-later time, and then when you get one piece of work you fire off that notification separately to every sleeping thread, including identical come-back-later times in each one.

toast03• 5 years ago

That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and your inbox is full enough for three inboxes. Does that fact, in itself, mean that you decide you're done for the day? No, it just means you have a much longer queue to work through than usual.

If my SLA is 24 hour response time, and the inbox is FIFO, and I can't drop old messages, I'm most likely not hitting the SLA. If they all came in overnight, I'll hit the SLA for day 1, but I will be busy all of day 2 and 3 and never respond on time. If after day 1, I get a days worth of messages every day, I'll never catch up.

c_o_n_v_e_x• 5 years ago

This reminds me of inrush current when starting large motors... You get a huge current spike when you initially turn on the motor, so large that it can trip the breaker.

One solution is to use a soft starter which slow brings the motor up to speed.

lookACamel• 5 years ago

That's not the thundering herd. If someone rings the door (request), only one person (agent, process) needs to answer the door. But what might happen is that everyone in the house rushes to answer the door. The people "thundering" to the door (and making a mess as they do so) are the "herd". This can quickly become a problem if there are a lot of people in the house and the doorbell keeps ringing.

toast03• 5 years ago

That's really not the traditional meaning of thundering herd, which is about waking up all the processes when a connection comes in, then they all try to accept it and it's a lot of work for nothing. You get much better results if only a single process is woken up for each event.

Your problem is a real problem though. Where I worked, we would call that backlog, and we would manage it with 'floodgates' ... When the system is broken, close the gates, and you need to open them slowly.

In an ideal world, your system would self-regulate from dead to live, shedding load as necessary, but always making headway. But sometimes a little help is needed to avoid the feedback loop of timed out client requests that still get processed on the server keeping the server in overload.

lmilcin6• 5 years ago

1M requests per minute on 1000 web instances is not an achievement, it is a disaster.

It is ridiculous people brag about it.

Guys, if you have budget maybe I can help you up this by couple orders of magnitude.

smashed• 5 years ago

To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing.

It seemed to be all about how to extract the most performance from the lemon they had to deal with.

I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/

lmilcin6• 5 years ago

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks.

Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes.

This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything.

I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing.

poor hindsight from original developer (co-founder)

Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less?

It is not poor choice, it is lack of competency.

You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.

luckycharms810• 5 years ago

This is a comically yet incredibly common engineering bad take. When you run a company there is only one question to answer, one north star - does it make money ?

yuliyp• 5 years ago

Knowing nothing else, it's hard to know if this is good or not. It's 16 requests per second. Are those requests something like "Render a support article" or are they "Give the user a ranked feed of what they should see on their home screen"? Is most of the logic run by the web server or some combination of app servers / backend services behind it? What kind of hardware does the web server have?

All of those would affect the answer, and would preclude being able to guarantee "up this by couple orders of magnitude"

lmilcin6• 5 years ago

Well, to give you an idea, I am working on an service that implements rather complex business process involving fetching data from multiple sources, parsing binary blobs with market data in proprietary format, saving results to a database and so on. And it does around 10k requests per second on a single relatively normal node (8 cores, 64GB ram, etc.)

And no, it does not require any special tricks. It is regular Java / WebFlux / REST / MongoDB backend service.

CPUs can do really a lot and if your node processes 16 requests per second on a multi-core machine then you are using billions of clock cycles and gigabytes of possible transfer to memory for a single request. Something is not quite right...

stu2010• 5 years ago

Interesting to see this. It sounds like they're not on AWS, given that they mentioned that having 1000 instances for their production environment made them one of the bigger deployments on their hosting provider.

If not for the troubles they experienced with their hosting provider and managing deployments / cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putting in the time to investigate these issues. I'd also love to see some actual latency graphs, what's the P90 like at 25% CPU usage with a simple Gunicorn / gevent setup?

ksec• 5 years ago

I was wondering that too, but there aren't that many common cloud provider that has 96 vCPU offering.

I am also wondering on 144 Workers, on 96 vCPU which is not 96 CPU Core but 96 CPU thread. So effectively 144 Workers on 48 CPU Core possibly running at sub 3Ghz Clock Speed. But it seems they got it to work out in the end. ( May be at the expense of latency )

mst5• 5 years ago

Assuming you're running a system where normal request/response handling blocks on database queries it's often optimal to have more workers than available cpu threads and 1.5x is a common rule of thumb to try first.

j4mie• 5 years ago

If you’re delegating your load balancing to something else further up the stack and would prefer a simpler WSGI server than Gunicorn, Waitress is worth a look: https://github.com/pylons/waitress

JanMa• 5 years ago

Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it's easily done when using HaProxy but I have never read about people using it. I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.

kvalekseev3• 5 years ago

It's standard way to connect things in UNIX and provides better performance. For example postgresql tcp+ssl is 175% slower than socket https://momjian.us/main/blogs/pgblog/2012.html#June_6_2012

lttlrck• 5 years ago

But domain sockets only work between processes on the same machine, why would SSL be used in that case?

mst5• 5 years ago

I do that every chance I can get.

At a guess, it's probably most loved by people picking old school simple architectures that aren't the sort of thing that goes viral.

kvalekseev3• 5 years ago

HAProxy is a beautiful tool but it doesn't buffer requests that is why NGINX is recommended in front of gunicorn otherwise it's suspectible to slowloris attack. So either cloubhouse can be easily DDOS'd right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don't mention that problem while recommend others to try and replace NGINX with HAPRoxy.

lddemi3• 5 years ago

1. HAProxy does support request buffering https://cbonte.github.io/haproxy-dconv/2.2/configuration.htm...

2. our load balancer buffers requests as well

kvalekseev3• 5 years ago

From HAProxy mailing list about http_buffer_request option https://www.mail-archive.com/haproxy@formilux.org/msg23074.h...

In fact, with some app-servers (e.g. most Ruby/Rack servers, most Python servers, ...) the recommended setup is to put a fully buffering webserver in front. Due to it's design, HAProxy can not fill this role in all cases with arbitrarily large requests.

A year ago I was evaluating recent version of HAProxy as buffering web server and successfully run slowloris attack against it. Thus switching from NGINX is not a straightforward operation and your blog post should mention http-buffer-request option and slow client problem.

polote• 5 years ago

I wouldn't be very proud of writing an article like that.

Usually engineering blogs exists to show that there are fun stuff to do in a company. But here it just seems they have no idea, what they are doing. Which is fine, I'm classifying myself in the same category.

Reading the article I don't feel like they have solved their issue, they just created more future problems

luhn3• 5 years ago

Unfortunately HAProxy doesn't buffer requests*, which is necessary for a production deployment of gunicorn. And for anybody using AWS, ALB doesn't buffer requests either. Because of this I'm actually running both HAProxy and nginx in front of my gunicorn instances—nginx in front for request buffering and HAProxy behind that for queuing.

If anybody is interested, I've packaged both as Docker containers:

HAProxy queuing/load shedding: https://hub.docker.com/r/luhn/spillway

nginx request buffering: https://hub.docker.com/r/luhn/gunicorn-proxy

* It does have an http_buffer_request option, but this only buffers the first 8kB (?) of the request.

Twirrim2• 5 years ago

Couldn't Apache httpd just do all of that for you? mod_buffer provides request buffering, and mod_proxy_balancer provides load balancing capabilities.

luhn3• 5 years ago

Can Apache do request queuing?

jhgg3• 5 years ago

This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP's post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless now that we're using envoy, so that'll probably go away soon.

Works amazingly well! We run our python API tier at 80% target CPU utilization.

lddemi3• 5 years ago

glad you are seeing such awesome performance with gevent+envoy! which part of our experience do you think is suspect?

jhgg3• 5 years ago

So, in guincorn default mode (sync), the mode I'm assuming you're using. This means you really have 1 process handling 1 request at a time. The "thundering herd" problem really only applies to connection acceptance. Which is to say, that in the process of accepting a connection, it is possible to wake all idle processes that are waiting for a connection comes in (they will wake and hit EAGAIN and then go back to waiting.) Busy processes that are servicing requests (not waiting on the accept call) will not be woken, since they aren't waiting on a new request to come in. The "thundering herd" problem as I understand it, can indeed waste CPU cycles, but only on processes that aren't doing much anyways. I do however believe that `accept()` calls have been synchronized between processes on Linux for a while now to prevent spurious wakeups. You should verify you're actually doing spurious wakeups by using `strace` and seeing if you are seeing a bunch of `accept()` calls returning EAGAIN.

In gunicorn, `sync` mode does exhibit a rather pathological connection churn, because it does not support keep-alive. Generally, most load balancing layers already will do connection pooling to the upstream, meaning, your gunicorn processes won't really be accepting much connections after they've "warmed up". This doesn't apply in sync mode unfortunately :(. Connection churn can waste CPU.

Another thing to also note is that if you have 150 worker processes, but your load balancer only allows 50 connections per upstream, chances are 100 of your processes will be sitting there idle.

Something just doesn't feel quite right here.

EDIT: I do see mention of `gthread` worker - so you might be already able to support http-keepalives. If this is the case, then you should really have no big thundering herd problem after the LB establishes connections to all the workers.

mac-chaffee• 5 years ago

Could the discrepancy be explained by the type of responses?

Sounds like an app like clubhouse might have lots of small, fast responses (like direct messaging), where very little of the response time is spent in application code. Does your API happen to do a lot of CPU-intensive stuff in application code?

jhgg3• 5 years ago

Our app is also a messaging app. So lots of small & fast responses.

trinovantes• 5 years ago

I've always used nginx for my servers. Is HAProxy that much better to consider learning/switching?

ram_rar• 5 years ago

Python's model of running N separate processes for your app is not as unreasonable as people might have you believe! You can achieve reasonable results this way, with a little digging.

I have been through this journey, we eventually migrated to Golang and it saved a ton of money and firefighting time. Unfortunately, python community hasnt been able to remove GIL, it has its benefits (especially for single threaded programs), but I believe the cost (lack of concurrent abstractions. async/await doesn't cut it) far outweigh it.

Apart from what the article mentions, other low hanging fruits worth exploring are

[1] Moving under PyPy (this should give some perf for free)

[2] Bifurcate metadata and streaming if not already. All the django CRUD stuff could be one service, but the actual streaming should be separated to another service altogether.

jstrong• 5 years ago

I read the article and could not believe that was their takeaway. sometimes people are determined to vindicate their technology choices, no matter what.

petargyurov• 5 years ago

Which exacerbated another problem: uWSGI is so confusing. It’s amazing software, no doubt, but it ships with dozens and dozens of options you can tweak.

I am glad I am not the only one. I've had so many issues with setting up sockets, both with gevent and uWSGI, only to be left even more confused after reading the documentation.

tarasglek• 5 years ago

Have to wonder how well haproxy works vs balancing by making gunicorn listen via SO_REUSEPORT and letting the kernel balance instead (ala https://talawah.io/blog/extreme-http-performance-tuning-one-...)

Reining in the thundering herd: Getting to 80% CPU utilization with Django

Related Stories