Why does one Nginx worker take all the load?

https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/

by porker • 9 years ago

164 49 9 years ago

blog.cloudflare.com

majke4• 9 years ago

Author here. There are three highlights in this blog post:

- epoll() seem to have LIFO behavior which can result in uneven load balancing across workers (in case of accept()ing from a shared socket)

- using REUSEPORT can worsen latency in a high-load case.

- blocking accept() on Linux is pretty buggy, you basically can't close() underlying socket

FBISurveillance• 9 years ago

Thanks for the post!

Is there anything you could suggest looking into to improve nginx processing of high volume large POST bodies (5KB, tens of thousands per second)?

I'm using kernel 4.13 with BBR congestion control on 20Gbps network and seem to hit this weird bottleneck when it does not matter how many nginx processes I have, it works similarly terrible on both 16-core and 64-core servers. (Of course irq/process affinity is in place which makes me think it's nginx issue.)

slackingoff2017• 9 years ago

At that speed it might not be Nginx. Check out Cloudflare's other blog posts too.

https://blog.cloudflare.com/how-to-receive-a-million-packets...

And thanks OP. I'm sure thousands of companies do these kinds of optimizations but CloudFlare is one of the only to bother sharing.

spydum• 9 years ago

Not an nginx user here, but I do know sometimes certain modules in Apache will add significant latency on POST payloads by waiting for the entire POST payload to complete before sending to the backend (in a reverse proxy setup). The idea is that if the backend fails, it can retry on the next node. For large payloads this sucks for latency.. No idea if nginx has this problem.

tyingq4• 9 years ago

You mention 5KB...is there some chance that it exceeds 8KB?

There's a setting called client_body_buffersize. (http://nginx.org/en/docs/http/ngx_http_core_module.html#clie...) It defaults to 8k on x86-64.

If you exceed that setting, it writes the body to a temporary file, which slows things down.

Maybe try bumping that up to 16k?

Also see the other related settings on that page.

Edit: Maybe also test out switching from urlencoded to multipart for the post? For some types of data, urlencoded can bloat the size quite a lot.

Twirrim2• 9 years ago

You kind of did a throwaway comment in the blog post about FIFO helping, but it wasn't clear to me why it would result in any different behaviour from LIFO. Wouldn't that mean it just hits the first host all the time instead of the last?

khc• 9 years ago

(work at cloudflare but not on the same team as the blog author)

I believe the FIFO/LIFO refers to the queue of processes waiting to receive a request. Since each process calls accept()/epoll(), then loops around and call it again, FIFO would ensure the one that just processed a request would go back to the end of the line.

eridius• 9 years ago

It hits the first worker, and that first worker leaves the FIFO queue, and thus is no longer the first worker anymore. When it calls epoll again, it'll end up as the last one instead.

cryptonector4• 9 years ago

Wow. This is a terrible state of affairs!

The LIFO behavior of epoll maybe could be worked around by having just one process get accept events at once and pass the baton to the next process when it accepts a client. I can't believe I have to think of such things. I have a daemon that uses epoll that currently scales quite well on one CPU for what it does, but which I'll probably have to make support multiple processes (or threads) at some point.

And then there's the fact that epoll is not fork-safe.

cryptonector4• 9 years ago

The other possibility is to have a thread in each process just to block in accept4(2), add it to a queue, and send an event to the other thread in the same process.

RickJWag• 9 years ago

Thanks for the nicely written article, especially the pictures.

Articles like this make for good reading.

foxhill2• 9 years ago

with regard to the epoll-and-accept method - why is the LIFO queue bad? sure, you're seeing more load on one core, but it must be finishing requests by the time it starts working on the next one - i.e not causing requests to stall as a result of the imbalance.

put that in the context of a multi-socket system, then core-caching and NUMA issues (even turbo clocking on an individual core) would seem to imply that LIFO is in fact exactly what you want, no?

majke4• 9 years ago

Imagine a HTTP server supporting HTTP keepalives. Say 10 new connections come in. Say on average they will all go to a single worker. Then say all of the connections request a heavy asset.

You will end up with one worker handling these 10 connections/requests and spinning at 100% CPU while other workers idle. I'm not saying this bad load balancing is a problem affecting everyone, it very much depends on your load pattern.

foxhill2• 9 years ago

http isn't particularly my thing, so if i'm wrong, please correct me! however, i was of the impression that a keep-alive session wouldn't return to the accept stage - the socket is still open, surely?

kevin_nisbet• 9 years ago

I find this very interesting.

As already pointed out, I imagine there could be some subtle benefit to delivering a majority of connections to a single process while it isn't overloaded, so even perfectly even load balancing can have some disadvantages.

majke, the benchmark you did with SO_REUSEPORT, was that against a NUMA system / recent kernel? I've completely lost track of it, but I think I remember reading that there has been some work on mapping the NIC Queues, Cores, and Workers together, allowing a flow to worker being able to be processed on the same core. I'm just curious if the benchmark shown was able to take advantage of that or not (and as an aside how much of a benefit mapping this together really has).

otterley4• 9 years ago

If the first worker isn't actually idle, then the scheduler will assign the next incoming request to an idle worker. Am I mistaken? If not, what's the problem here?

zerop• 9 years ago

For more interest, read Nginx architecture on "The architecture of open source Applications" www.aosabook.org/en/nginx.html

_Codemonkeyism• 9 years ago

Why doesn't NGINX implement work stealing? Wouldn't that help?

tyingq4• 9 years ago

You can control of the queue, but not the entries. You can pass the whole queue/socket around via sendmsg(), but not single entries in the queue. This is hard to solve well in user space.

wahern2• 9 years ago

You can accept a socket and pass it to worker processes/threads using whatever scheme desired.

derefr• 9 years ago

So, what would a non-userspace solution look like? An HTTP keepalive + pipelining + HTTP2 implementation in the kernel that forwards the demuxed messages as separate packets onto a specially-typed socket, such that a prefork daemon can accept(2) individual HTTP request packets?

cat199• 9 years ago

soo..

anyone have any insight on this behavior on other systems? (e.g. FreeBSD kqueue)

barrkel3• 9 years ago

IOCP on Windows would use a thread pool to dispatch work after async sockets come due for some action. And after work for any given socket is completed, it's possible for work to start on a different socket's completion, with the "context switch" (or rather, continuation callback) happening in userland, rather than requiring a context switch. This inversion of control acts a bit like work stealing.

majke4• 9 years ago

Comment from reddit: https://www.reddit.com/r/programming/comments/78f1if/why_doe...

LIFO is also the strategy for Completion Ports in Windows:

Threads that block their execution on an I/O completion port are released in last-in-first-out (LIFO) order, and the next completion packet is pulled from the I/O completion port's FIFO queue for that thread. This means that, when a completion packet is released to a thread, the system releases the last (most recent) thread associated with that port, passing it the completion information for the oldest I/O completion.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

citrin_ru2• 9 years ago

On FreeBSD with kqueue and without accept mutex workers load is also uneven (it became more even when you have more load). With accept mutex load is more even but using mutex adds some overhead.

citrin_ru2• 9 years ago

It looks like nginx ability to accept several connections in a row can improve throughput, but increases latency. In DragonflyBSD now it is possible to tune "batch size": http://lists.dragonflybsd.org/pipermail/commits/2017-October...

bogomipz2• 9 years ago

I have a question about the following passage regarding the LIFO nature of epoll-and-accept:

"This behavior causes the busiest process, the one that only just went back to event loop, to receive the majority of the new connections."

What is meant by a process "that only just went back to the event loop"? The worker process is the event loop no? Isn't the worker process never not running an event loop?

Or is this just meant to say when the worker process is not in the section of the event loop that's responsible for the enqueueing and dequeueing of events?

dcow• 9 years ago

At the end of the event loop you wait for new events. That part.

bogomipz2• 9 years ago

Thanks. For clarification, the event queue that nginx's worker process is using for its input is the same accept() queue that the passive nginx listener socket is populating. Is that correct?

dboreham2• 9 years ago

Somewhat amazed that this stuff _still_ isn't properly sorted out given that I was working on applications with a keen interest in the problem in 1996!!

NT's IO Completion Ports implemented some kind of CPU-balanced work delivery mechanism from the beginning, presumably inspired by Cutler's previous work on VMS's $QIO. AIX afaik also had something similar. Linux not so much...but in the intervening 20 years I had assumed epoll() was fixing this.

cryptonector4• 9 years ago

You assume things in Linux-land are "designed". It's more like natural evolution. It's very organic. As you'd expect, there's a lot of decomposition around. It kinda smells as you'd expect.

The Linux community is known to be infected with a drug-resistant strain of NIHV -- the Not Invented Here Virus. This means that if there's a thing like kqueue, or NT I/O Completion Ports, or Solaris Event ports, and Linux could copy and/or improve one (or more) of those, then you can count on Linux to invent a new thing that isn't as good as any of the others.

Perhaps some pharmaceutical will someday work on a treatment, cure, and/or vaccine for that strain of NIHV. More likely, the community will someday evolve resistance to NIHV, or else make NIHV moot by driving out all the competition.

The good news is that you can resist NIHV yourself. All you have to do is not believe that you alone can solve mankind's problems in a vacuum.

dboreham2• 9 years ago

I don't remember it quite like that in this specific case. Rather I heard that there was push back against implementing a completion port model from somewhere (Redhat legal??) fearing patent action by MS. IBM had a mutual license agreement with MS I believe, which explains why AIX has something similar. The patents in question may even be in the set litigated over in the wake of Cutler moving to MS from DEC (which MS paid $$$ for and hence might be inclined to defend aggressively).

Why does one Nginx worker take all the load?

Related Stories