One Million Concurrent TCP connections

http://blog.whatsapp.com/index.php/2011/09/one-million/

by pvodsevhcm • 15 years ago

192 65 15 years ago

blog.whatsapp.com

getsat2• 15 years ago

What is the kernel structure overhead (in bytes) per connection on FreeBSD?

jsr• 15 years ago

OK, you hooked me with the title. But "FreeBSD + Erlang" was kind of a dissatisfying reason for how you achieved it. Would love to hear more details! How far we've come since http://www.kegel.com/c10k.html

getsat2• 15 years ago

They could have done it using C, Ruby, Python, or any other language. kqueue is what makes FreeBSD (and OSX) awesome at concurrency.

http://en.wikipedia.org/wiki/Kqueue

silentbicycle• 15 years ago

How does kqueue compare to epoll on Linux? I've written C code using kqueue on OpenBSD and OS X, but have only used epoll via libev (and not at especially high load). I thought the big change came from trading level- for edge-triggered nonblocking IO, but maybe the kqueue implementation is superior for sockets somehow?

The main advantage Erlang has over C/Python/Ruby/etc. is that asynchronous IO is the default throughout all its libraries, and it has a novel technique for handling errors. Its asynchronous design is ultimately about fault tolerance, not raw speed. Also, it can automatically and intelligently handle a lot of asynchronous control flow that node.js makes you manage by hand (which is so 70s!).

You can make event-driven asynchronous systems pretty smoothly in languages with first class coroutines/continuations (like Lua and Scheme), but most libraries aren't written with that use case in mind. Erlang's pervasive immutability also makes actual parallelism easier.

With that many connections, another big issue is space usage -- keeping buffers, object overhead, etc. low per connection. Some languages fare far, far better than others here.

asomiv3• 15 years ago

The kqueue interface is awesome, but unfortunately kqueue is bugged on OS X: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AN... (as are poll() and select()!)

indygreg2• 15 years ago

Technical details would be interesting. Until then, here's Urban Airship's post from last year on 500k connections on Linux:

http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-...

chubs2• 15 years ago

They beat me to it! I've only gotten to 500k on EC2, however i believe there's some trickery in their firewalls / NAT which is holding me back... If anyone's interested in the gory details, see: http://splinter.com.au/tag/comet

forsaken• 15 years ago

Curious about what you've seen from the EC2 networking gear that's holding you back? Firewall not letting more connections through?

chubs2• 15 years ago

Well, at the moment i can't pin it down to EC2, but it's the only thing i can imagine it'd be. The network/cpu/memory usage is all healthy, and there's nothing in the kernel log, so that's what i'm guessing is the cause. Although i may be wrong.

mrb• 15 years ago

I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile an optimized version of the p2p app (to allocate data structures as small as possible for each client) to support up to one million concurrent TCP connections. It wasn't a test system, it was a production server routinely reaching this many connection at its daily peak.

If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.

One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.

I cannot find a link to my story though...

gtani• 15 years ago

(Not your link but) some pointers on tuning erlang/OTP servers:

http://www.erlang-consulting.com/thesis/tcp_optimisation/tcp...

http://www.trapexit.org/Building_a_Non-blocking_TCP_server_u...

http://groups.google.com/group/erlang-programming/browse_thr...

(this erl mailing list thread is pretty typical, if you put up code, describe your app, hardware, network, database/external dependencies, etc, you'll get a ton of good advice about killing off bottlenecks. Another example

http://groups.google.com/group/erlang-programming/browse_frm...

mmaunder• 15 years ago

Agree with this being underwhelming. Maybe they'll hire someone who understands that Erlang is using kqueue to make this possible.

Running netstat|grep like this on a high concurrency server takes a long time to run. I've never found a faster way to get real-time stats on our busy servers and would be interested if anyone else has.

derobert• 15 years ago

You might be able to parse /proc/net/tcp yourself faster, especially for just counting by state.

thomasknowles2• 15 years ago

Actually I would run ss over the netstat for such high concurrency connection as netstat hits /proc/net/tcp and it's quite slow. However, you need to have the tcp_diag module loaded otherwise it falls back to /proc/net/tcp.

adient• 15 years ago

You may have tried but generally add -n to netstat to speed it up by just displaying IPs instead of trying to resolve them. I've never seen a system with so many concurrent connections hands on so there could be some other bottleneck.

Rickasaurus2• 15 years ago

This may be a dumb question (I'm not a networks guy) but how do you maintain so many connections with just 65535 ports? Can you have more than one connection per port?

pacala2• 15 years ago

There is one unique connection per (address, port) pair.

Rickasaurus2• 15 years ago

Ahh I see, you just give one box a ton of IPs. Thanks :)

huhtenberg2• 15 years ago

Actually, the quintet of [protocol, src_ip, src_port, dst_ip, dst_port] is what's unique (protocol being TCP or UDP)

caf3• 15 years ago

The server generally only ever uses one port, no matter how many clients are connected. It is the tuple of (client IP, client port, server IP, server port) that must be unique for each TCP connection - so the limit of 65535 ports is only relevant for how many connections a single client can make to a single server.

richardw• 15 years ago

You can have multiple connections on the same port from the same client. E.g. all connections via HTTP commonly go over the same port and a single browser usually has 2-4 simultaneous connections to the server. They don't use multiple ports to achieve this.

nathanappere2• 15 years ago

I believe this is incorrect. The server usually listen on one port, but everytime it does an accept, a different random port is used, and the client start talking to the server on that new port.

huhtenberg2• 15 years ago

In absolute numbers - wow, that's impressive.

It was several years ago, but I've done my share of high-concurrency stuff under Linux and the highest I got to was about 200K connections - at which point the single-threaded server bottlenecked at its disk I/O.

The main issue is not the actual connection count, it's what the per-socket OS overhead is (so not exhaust non-swappable kernel memory), how many sockets are concurrently active (have an inbound or outbound data queued) and if the application can handle all the events that epoll/kqueue report. This is not a rocket science by any means, and the kernel is relatively easy to fine-tune even when the actual load is present.

tworats• 15 years ago

The WhatsApp guys are very sharp ex-Yahoo guys who've had tremendous experience with scaling systems. Rick Reed is fairly legendary. Yahoo was a long time FreeBSD shop, so it's not surprising they went with that.

I hope they publish how they did it - in fact let me drop them an email and see if I can convince them to do so.

cperciva• 15 years ago

Yahoo was a long time FreeBSD shop

FWIW, Yahoo still uses FreeBSD extensively.

lenn0x• 15 years ago

It's been done before. Here is an article using Erlang and Linux.

http://www.metabrew.com/article/a-million-user-comet-applica...

Part 3 is my favorite. http://www.metabrew.com/article/a-million-user-comet-applica...

jen_h3• 15 years ago

This article series is kind of a Bible to me. It's not going to solve all of your problems, for sure, depending on yer stack, and setting yourself up for a forkbomb isn't the wisest in all situations, but it's got a lot of good advice & is pretty good about providing you a "Okay, tweak these parameters and then try to break it" baseline.

jen_h3• 15 years ago

A. That is AWESOME. Props, guys! B. Just how long did it take for that netstat to return? ;)

spokengent2• 15 years ago

FWIW, netstat isn't fun to use. conntrack is better, or cat /proc/net/ip_conntrack

Luyt• 15 years ago

They're on FreeBSD. conntrack is a Linux utility.

KonradKlause• 15 years ago

conntrack is not netstat. I requires netfilter's connection tracking feature.

Using conntrack on a 1MC system will waste even more kernel memory!

imsy• 15 years ago

Good work. On Imsy (www.imsy.com) we hope to achieve the same with Node.js on EC2. Presently the numbers are smaller (in 100K range).. but looks like it will scale smoothly till 1 million.

henry501• 15 years ago

It's hard to say that something is going to scale smoothly up an order of magnitude until you're there. 900K requests is a lot of room for things to go wrong.

nivertech2• 15 years ago

Node has memory limit of about 140K connections. I was able to accept and actually do something usable with 1M on EC2 and 3M on physical server (Erlang and Ubuntu/CentOS)

cnlwsu• 15 years ago

I would be curious about the hardware used, it can make it much more or less impressive. I have done a test with 1 million concurrent tcp connections using java (mina) on an Ubuntu system... but it had 64 gb of ram. It kept running for weeks under the load which I felt pretty good about.

gabi38• 15 years ago

How would kqueue compare to Windows's IO completion ports in terms of performance?

flzz• 15 years ago

Just because you can doesn't mean you should.

nivertech2• 15 years ago

Not many can, but some fortunate enough to actually need this stuff.

kitsune_• 15 years ago

How much ram does this single server have, what about its cores?

One Million Concurrent TCP connections

Related Stories