Improving Linux networking performance

by sciurus • 12 years ago

235 48 12 years ago

fideloper• 12 years ago

So...we're all upvoting this hoping someone else understands networking at this level, right? And that maybe they'll do something awesome with it.

lucb1e• 12 years ago

I probably couldn't improve the actual code myself, but I do understand what is being talked about conceptually. I enjoyed the article and learned a bit from it, thus I upvoted.

wmf4• 12 years ago

I understood this article and it is relevant to my interests since 25G NICs are coming this year.

agrover• 12 years ago

source? I thought the next step was 40G?

alricb• 12 years ago

Possible contrast & compare: the presentations on OpenBSD's network stack at http://openbsd.com/papers/

scott_karana• 12 years ago

The first relevant-seeming presentation is from 2009[1] and doesn't really get into the pitfalls of low-latency switching/handling like this article does.

I'm definitely interested to see how other operating systems handle this, though. In particular: Windows (is networking in user-mode?) and Solaris-likes.

1 http://quigon.bsws.de/papers/2009/eurobsdcon-faster_packets/

joshbaptiste• 12 years ago

Indeed, there has been a large amount of "bypass the kernel" campaigns in the last couple of months. Robert Graham's 2013 Shmoocon talk has a great introduction into the whys and hows of this movement.

https://www.youtube.m/wco?v=D0atch9jdbS6oSI

Facebook had a job posting that showed up on HN for a position to help speed up Linux's networking stack. While I doubt these improvements will surpass the kernel bypassing model, I'm glad a developer has decided to tackle this head on and help overall efficiency.

adamnemecek• 12 years ago

here's the fixed link

https://www.youtube.com/watch?v=D09jdbS6oSI

voltagex_• 12 years ago

Direct video link: http://www.shmoocon.org/2013/videos/Shmoocon%202013%20-%20C1...

SEJeff• 12 years ago

Look no further than Solar Flare's Open Onload[1], Mellanox's VMA[2], Myricom's DBL[3], Intel's DPDK[4], etc to see that kernel bypass tech is used in 10G+ networking all the time.

It is super fast, but also lower latency. Many of the features used in lower latency environments like busy polling have been slowly making their way upstream[5]. I think Linux will get pretty close to the userspace kernel bypass stuff, and this is coming from a guy who works in a low latency environment for $REAL_JOB.

[1] http://www.openonload.org/

[2] http://www.mellanox.com/page/software_vma?mtag=vma

[3] https://www.myricom.com/support/downloads/dbl.html

[4] http://www.intel.com/content/www/us/en/intelligent-systems/i...

[5] https://lwn.net/Articles/540281/

fit2rule• 12 years ago

Another great example is snabbswitch, which combines a lot of really great technologies to provide superlative performance:

https://github.com/SnabbCo/snabbswitch/wiki

Hypervisor, Kernel, LuaJIT, Userspace. Amazing!

TheCondor• 12 years ago

There is a lot more than bypass the kernel here. You need special dma hardware that will preload cpu cache lines and stuff like that. Maybe even specialized cpu cache that can be pinned

mobiplayer• 12 years ago

It seems to be pretty uncommon for Linux setups to have, e.g. TOE enabled, but in my humble opinion it is an easy win on performance on 10G networks.

wmf4• 12 years ago

TOE isn't supported under Linux and it was only a win if your TCP stack was slow (and by "your" I mean Windows). TSO is enabled by default in Linux and is indeed an easy win.

arca_vorago• 12 years ago

Am I the only one that thinks we need to start at re-evaluation of BSD sockets first? I know Apple tried and gave up, but it just seems like so many of the building block pieces we use everyday could really use a major polish or some good competition.

wmf4• 12 years ago

This article is mostly about the lower parts of network stack like QoS and talking to the NIC; the user API is kind of orthogonal but equally important. There have been several research projects about improved networking APIs; my favorite is IX which gets line-rate performance while retaining kernel/user protection. https://www.usenix.org/conference/osdi14/technical-sessions/...

jalcazar• 12 years ago

This reminds me of MegaPipe. Basically it creates a pipe between kernel and user space. It uses batching too https://www.usenix.org/conference/osdi12/technical-sessions/...

trentnelson• 11 years ago

It bugs me that the source isn't available for stuff like this. Makes it tough to objectively evaluate things.

fulafel3• 12 years ago

Too bad improvements in network technology haven't found their way to consumer level. It's probably related to the stagnant broadband speeds, last mile bandwidth improvements slowed to a crawl many years ago and concurrently device connectivity actually moved to a slower networking tech, wi-fi. Now people are happy to use wi-fi for home desktop computers since their last mile net connection is so slow anyway.

It's 10 years since motherboard integrated 1G became commonplace in regular PCs, same for 10G is nowhere in sight...

mcpherrinm2• 12 years ago

10 gigabit over Cat6A cable is also power hungry and very tricky: It's approaching the limits of what you can do over that sort of cable. Plus the frequencies involved make the adapter design a lot more troublesome. I suppose we'll start seeing cheap 10 gig knockoff nics eventually, but it's not even everywhere in the datacenter, never mind the home.

hkhkhhjhnbv• 12 years ago

yep. Fiber is the way to go when you need 10Gbit. I was looking in to doing my place with Cat6 or 6a. Not only is 6a incredibly thick (thus, more difficult to run and probably terminate) but you need special tools and knowledge just to make sure the wiring is correct and there are no RF issues going on. It becomes way more about RF crosstalk and science, and less about simple wire connectivity.

skuhn2• 12 years ago

10 gig still isn't even commonly available on server motherboards, because of power / space / cost. There also aren't many copper 10 gig top-of-rack switches, just the Cisco Nexus 3064T and Arista 7050T come to mind. Juniper doesn't even have one.

It's easier for a lot of places to use twinax with 10 gig SFP+ switches rather than going copper 10 gig. That is definitely not going to trickle down to the consumer level.

It will probably be another 1-2 years before 10 gig is ubiquitous at the server level, and another 2-3 years after that before it is commonly on consumer equipment. Or maybe it never will be, and things will go in another direction.

donavanm5• 12 years ago

There also aren't many copper 10 gig top-of-rack switches, just the Cisco Nexus 3064T and Arista 7050T come to mind. Juniper doesn't even have one.

I might be missing your definition of tor. The juniper QFX5100 series has the 48T which does 48x 10GBASE-T plus 6x QSFP. the 5100-96S does 96x SFP+ and 8x QSFP. There are plenty of other cheap merchant silicon platforms that look similar. Personlly Im happy with DAC on SFP+ ports.

justincormack2• 12 years ago

Here you are, motherboard integrated dual 10G from Supermicro [1]. Around £400. They should be commonplace this year I think, it is just starting.

[1] http://www.supermicro.com/products/motherboard/Xeon/C600/X9S...

fulafel3• 12 years ago

Yeah, they exist in a select few workstation class Xeon boards. Intel announced those (server-targeted) chips in 2012, they take time to show up even in this high end niche. As there haven't been any announcements about mass market parts, it would be a small miracle if they showed up this year.

pixl97• 12 years ago

Yea, with the speed of SSDs increasing so quickly the speed at which a desktop computer can transfer a file and store it has greatly exceeded what gigabit can provide. At the SME level 10G is still too expensive to justify at this point for regular workstations.

greggyb• 12 years ago

So, a local ISP has recently released 10Gbit fiber to home (for a tiny subset of the City).

Would any consumers be able to use this, or would it only make sense with a commercial switch (probably for a shared connection for an apartment complex or similar?)?

What would I have to do at home to take advantage of it?

koffiezet• 11 years ago

Last mile speeds for end-users aren't that important, most networking interaction they require goes to the internet anyway. Internet providers in the USA trying to push 5Mbit to be classified as "broadband" should tell you enough... I'm used to full speed 100Mbit internet (actually 160, but my router only has a 100Mbit connection), but for average useage - the difference between 50 and 100Mbit is rarely noticeable - unless you download big files.

Once 4k or 8k streaming becomes more common, sure then you'd like to have full Gbit where max bitrate would be 340Mbit - but for even decent full-hd streaming - nobody needs more than 50Mbit (in real throughput that is).

Higher speed things are more important on servers that have to handle more and more connections, and the end of this horizontal expansion is not in sight. Everybody starts using the internet for more and more things, and this all has to end up on servers that have to handle connections from millions of clients.

zelos• 12 years ago

I love reading articles like this: "So, for example, a cache miss on Jesper's 3GHz processor takes about 32ns to resolve. It thus only takes two misses to wipe out the entire time budget for processing a packet."

Then I go back to adding another layer of abstraction to my bloated Java code and die a little inside.

gretful• 12 years ago

yeah, it's easy to cry when you see a real artist working on something, and you have to go back to your etch-a-sketch.

npsimons• 12 years ago

Real art may inspire the soul, but road signs keep you from dying.

eklavya• 12 years ago

Actually guys like him are making our life easier and so we don't have to worry about a lot of stuff like this. Abstractions are a necessity, don't feel bad. There is no way a small team using Java/Scala can accomplish what they usually do in the time frame they do it, without those abstractions. Computers are there to ease our lives after all.

p.s. not undermining the need to optimize and all

fulafel3• 12 years ago

For completeness, generally memory access that misses the last level cache incurs 60-110 ns latency on recent DDR3 based x86 hardware, see eg. http://www.sisoftware.co.uk/?d=qa&f=ben_mem_latency&l=en&a=

I don't know what exactly the 32ns measurement is from, sounds similar to the "in-page" figures on the above page.

icantthinkofone• 12 years ago

The first they should do is do what Facebook is doing and turn to FreeBSD for ideas in their attempt to make Linux as good as FreeBSD's: http://www.theregister.co.uk/2014/08/07/facebook_wants_linux...

riffraff• 12 years ago

IIRC netmap was a cool concept born on freebsd but also available for linux.

http://info.iet.unipi.it/~luigi/netmap/

arjn• 12 years ago

Interesting article. Reminds me of my time in grad school :-)

Looks like jasper's recommended way is to improve or find a way to bypass the memory managment (slab allocation) subsystem.

There should be a way to tack on a more network optimized memory management layer or allocator onto the regular one.

Could turn out be a good research project.

xenadu02• 12 years ago

Why are we still using 1500 byte packets at 100G again? Seems like there won't be any tricks left to make 1000G work. Does that count as technical debt?

donavanm5• 12 years ago

Pretty much everything has supported 9k jumbos for over a decade. The internets mostly 1.5k MTU, but you normally arent doing multi gigabit streams over public connectivity. The other argument is TSO. Your kernels probably writing a 64K "packet" to the NIC driver. When segmentation etc is handled by the hardware why do you care about the MSS? On the network device side the SerDes are the issue. And were already running parallel lanes there; 40 is 4x 10 lanes and 100 is 4x 25 lanes. Why not 10x 100 in a couple years?

nitrogen2• 12 years ago

When segmentation etc is handled by the hardware why do you care about the MSS?

Because of Ethernet's mandatory minimum inter-packet spacing.

Improving Linux networking performance

Related Stories