How to achieve low latency with 10Gbps Ethernet

https://blog.cloudflare.com/how-to-achieve-low-latency/

by jgrahamc • 11 years ago

142 39 11 years ago

blog.cloudflare.com

rasz_pl• 11 years ago

step one should be 'we move drivers to userspace and bypass kernel completelly'

edit: finished reading, its the last step :/ and ~halves best case they were able to achieve with tweaks.

baruch• 11 years ago

That's still not easy enough to do. UDP is simple enough but doing TCP in those drivers is still a pain. There are DPDK with LWIP or with some BSD tacked on top but so far I didn't find something that I can just take for a ride in that area.

For the extreme cases you really want to control the packet allocation for receive and make the buffers your app uses be also used for the network transmit so you get a real zero-copy.

ghshephard• 11 years ago

It's an absolutely wonderful article - perhaps one of the best I've seen regarding latency tweaking on a linux stack - but I don't understand why so little about the final step, "Since we have a Solarflare network card handy, we can use the OpenOnload kernel bypass technology to skip the kernel network stack all together:" was discussed. That was like the major punch line of the entire article, and it was given a minor "oh, by the way, " sort of treatment.

Great article regardless. I'm wondering what the latency would have been if they had started with that step.

thrownaway24243• 11 years ago

What are the tradeoffs? Is there any downside to this userspace network stack, or is it pure win?

Edit: other than needing the thousand-dollar NIC, I mean.

jnordwick• 11 years ago

No more local tcp dump. No more kernel routing, copying, logging, or filtering of packets. As someone who's hacked on OoL and other kernal bypass stacks a little, it is a huge win if you are latency sensitive especially for just thrashing the core by constant polling for absolute lowest latency.

usefulcat• 11 years ago

Having used it, it will generally consume all available CPU cycles on the core(s) on which it runs. So presumably it is doing a lot of busy-waiting. Which, if low latency is your top priority, may be just fine.

pjc50• 11 years ago

It's open-source but patent-encumbered, which is a strange state. It's not a user-space TCP stack but a "card-space" one: it's offloading all the work into the card.

jandrewrogers• 11 years ago

Kernel bypass for networking and disk I/O (common in good databases) with userspace implementations has one large downside: it requires an enormous amount of sophisticated software engineering to access the integer factor gains in performance possible.

This requires the creation of a lot of code to reimplement the kernel functionality you need by a software engineer of atypical skill to ensure a robust and performant outcome. In other words, it is expensive and requires a skill set that is relatively rare to use well. However, if you have significant server CapEx and OpEx it can totally be worth it to reduce your required server footprint.

BTW, you can do this with many common Ethernet chipsets, including the ubiquitous Intel ones, but they are all a bit different to work with.

kasey_junk• 11 years ago

In my experience there is a pretty high operational burden to it. You spend a lot more time configuring, testing, diagnosing errors, etc.

scurvy3• 11 years ago

There are a lot of applications where it's a huge win. Security applications (snort, bro, flow monitoring) leverage onload/bypass with multiple receive queues to massively scale up.

Create a receive queue for each core on the NUMA node, spin up parallel processes (1 for each core and pin to the core), then aggregate the data. Wire rate monitoring with a $700 NIC and a commodity PC. Pretty amazing considering what companies used to pay for such performance.

thrownaway24243• 11 years ago

There have been a spate of articles lately that sum up to one fact: out of the box, all Linux distributions are tuned for nothing in particular. With some configs you can reduce your latency by an order of magnitude, increase your MySQL ops by an order of magnitude, etc. This is one reason why I, as an application-level programmer, am glad to work at a company big enough to have someone else dealing with the platform on my behalf. I don't want to spend ten years slowly discovering traps like default window size tuned for T1 speed, default min_rto tuned for interplanetary traffic, not enough PIDs to support a 96-core machine, etc.

fulafel2• 11 years ago

Nitpick: Maybe you are thinking of Windows wrt buffer sizes? Linux TCP out of the box has window scaling enabled since 2004 or so. "sysctl net.ipv4.tcp_rmem" tells me max buffer size is 6+ MB on my old 4GB laptop, which should be good to saturate 10 a gigabit pipe with up to 50 ms RTT... Mininum retransmission tuning might get you something on your own network if it's congested but would probably be bad to crank down for general internet usage, given things like bufferbloat and large delays caused by wireless L2 retransmissions.

acd• 11 years ago

Linux TCP window tuning is in fact off, agree with thrownaway. It's by the way a trick that the CDN providers use since they tune their stacks to increase the download speed. But if you could as well do the same tuning yourself.

superuser2• 11 years ago

How can one learn to do this? Are there good books/classes on the topic? Learning from mentors seems pretty vulnerable to cargo culting/bad science but I don't know where to even start.

j1e• 11 years ago

http://www.brendangregg.com/

rdtsc3• 11 years ago

all Linux distributions are tuned for nothing in particular.

Which is good, because often tuning means making trade-offs. A lot of those things like polling and interrupt shuffling have side effects and might for example negatively impact other scenarios -- presumably not everyone out there is running single client-server ping-pong with small UDP packets.

Yes a lot of parameters have retained default values from back in the day when system characteristics were different. Switching defaults drastically might break existing code that relied on them. Perhaps those are better updated during major distro version upgrades (RHEL 6->7 for ex).

In general though, I would say Linux is tuned for average throughput not latency. Sometimes they go hand in hand, sometimes they don't. Over the years there have been patches and forks related to operating on low latency environments (stock trading, media processing, hardware control, graphics in desktop environment). Those range from parameters tweaks in /proc to kernel patches, to almost full kernel bypass via things like DPDK for Ethernet for example.

thrownaway24243• 11 years ago

Sure, they are all tradeoffs of some kind or another. It's crazy to think that one person deploying a linux box on AWS would be aware of them all, though. For example what is the default file-max on Ubuntu? Why?

billyhoffman• 11 years ago

I agree, and Linux's default state is a matter I raise when people talk to me about how awesome DevOps and Full Stack developers are. They may be able to launch a VM and get a web stack up and running with yum, but that's really not reasonable for production usage

exelius2• 11 years ago

The way we work this is that DevOps builds the systems, and the performance testers tune the configs based on their testing. Performance testing is really not the right word for what they do; it's really performance engineering.

Regardless of whether you use DevOps or not, if scalability is an issue, someone needs to be doing performance testing and sizing to have a rough idea of how many users per node you can support. It's part of the QA cycle; and that doesn't go away with DevOps.

exelius2• 11 years ago

Yeah, you forgot the default file descriptor limits that are an order of magnitude too low for running any kind of modern web service. At least it doesn't require a patch and recompile anymore like it used to back in the 90s when I was running big IRC servers.

scurvy3• 11 years ago

Isn't this what tuned is for?

http://servicesblog.redhat.com/2012/04/16/tuning-your-system...

fulafel2• 11 years ago

The extreme measures (bypassing kernel etc) allowed them to shave ~20 us off the latency, which corresponds to a few kilometers. So this could indeed make a difference for latency-bound apps talking to eachother inside a data center.

wtallis2• 11 years ago

Really important point: this is measuring baseline latency of sending one packet at a time. They're not measuring latency under load where queueing delays are possible. The picture there usually looks pretty different and some of the settings they're changing here may be harmful or irrelevant to that case.

majke• 11 years ago

Totally. Throughput is an enemy of latency. If you have some other applications on that RX queues or CPU's you can't expect any latency numbers.

brianwawok2• 11 years ago

Also why upstream smart batching is so important. If you only have 1 packet shovel it off. But maybe wait 1-2 micros and see if you can combine 10 packets (up to MTU) to decrease overall latency.

wtallis2• 11 years ago

It depends some on what layer you're talking about. Nagle's algorithm makes a lot of sense in a lot of situations. But if you've got a complete packet that's ready to encapsulate in an ethernet frame and send over the wire, you only want to try for batching if you're actually hitting a CPU limit from interrupt frequency or something like that. Unless you're dealing with WiFi where extreme amounts of packet aggregation are necessary to get close to the theoretical throughput.

StillBored2• 11 years ago

NUMA pin the DMA buffers too. I didn't see that in the article. With the app I worked on getting the PCI bus/CPU/memory/interrupt all pinned to the "right" node gave us a good 30% performance improvement.

Put another way, identify which socket the PCIe bridge and device are attached to, then make sure the DMA buffers, and interrupt handler are pinned to that node.

Also, don't touch the buffers from the other socket. Strangely enough touching the DMA buffers from the other node just once was enough to cause a performance fall off.

monocasa• 11 years ago

Probably because the cache line changes from Exclusive to Owned in MOESI.

StillBored2• 11 years ago

Yah, that is the obvious choice, but it wasn't as clear cut. Especially with the behavior we were seeing where the performance problems would remain long after the lines were flushed from the CPU caches. I spent a fair amount of time looking at Intel docs on snoop filters and the like and never really satisfied myself that I knew exactly what was causing the problem.

wyldfire• 11 years ago

Yeah, with previous generations of chipsets/CPUs, I couldn't saturate quad-10GbE receive streams unless I bound the buffers to particular nodes and IRQs to corresponding cores.

hpcadmin007• 11 years ago

10 GbE is often good enough for most use-cases. For same datacenter same-floor gear and if there is a pressing requirement for low latency, deploying 1-2 generations old Infiniband (IB) gear is likely a smarter approach because the cost point approaches 10 GbE and the latency is sub-microsecond-scale (0.5-0.7 us vs 20-30 for 10 GbE, which is a factor of 40-60x). Deploying OFED can be a pain unless using a distro like Rocks, don't mix-match different vendors' gear (stick to all Mellanox or all QLogic) and be sure the specific bios versions of everything are supported. For Linux Rocks clusters support / deployment, we've used the cool & helpful folks over at StackIQ (was Cluster Corp).

How to achieve low latency with 10Gbps Ethernet

Related Stories