Why upgrading your Linux Kernel will make your customers much happier

http://samsaffron.com/archive/2012/03/01/why-upgrading-your-linux-kernel-will-make-your-customers-much-happier

by sams99 • 14 years ago

260 55 14 years ago

ChuckMcM4• 14 years ago

TL;DR version - another new guy discovers TCP slow start, and not doing it, wow major speedup! Now, imagine the whole Internet doing that, whoops.

I once peevishly pointed out that if you weren't required to stop at stop signs your commute would go faster, but if nobody was required to stop at stop signs it would be slower because every other intersection would have an accident blocking the way.

This is also very much true of TCP congestion control algorithms. And while a few people not using them can get away with it, everyone not using them and you will find your network latency goes from a median with a low standard deviation, to a slightly lower median with a HUGE standard deviation.

One of the things that slow start does is it spreads the change in median latency over a longer period of time. You can think of this intuitively where each new connection starts slow and then gradually gets faster, until it is as fast as it can be, and as more people start connections they start slow and get faster, while the current connections get slightly slower to accomodate the new traffic. The result is a non-chaotic adjustment of the network flow.

The converse is that everyone starts out going as fast as they can, they not only overwhelm the node the node ends up getting massively congested for a moment trying to sort things out. And of course IP doesn't care if you lose a fragment, you'll eventually resend it. So now during this massive congestion the re-transmits are causing more congestion. You get lots of pushback and finally everyone is back to a level where the network is doing ok with it and wham! a new connection opens up and everyone gets hosed again and backs off again, and then ramps up again.

Moral of the story, if only you don't do slow start you can be fast, if everyone doesn't start slowly, the network latency gets really unpredictable and poor.

ajross2• 14 years ago

It's more complicated than that though. Slow start is tuned for long-lived pipes sending large files, which doesn't match the observed load. And in any case the congestion bottleneck on modern networks is almost always in the last mile to the client anyway: most web services can freely pump as much data into the network as they want[1].

[1] Jim Gettys and his bufferbloat posse might have something to say to you though.

wmf• 14 years ago

Yes, I think those bloated buffers can easily absorb 10-packet bursts. If bufferbloat gets fixed there may be packet loss in the initial window burst and people should adjust window sizes back down. But I wouldn't worry about less-slow start unless it actually causes packet drops.

ChuckMcM4• 14 years ago

This is something that I think is slowly going to bite us big time. So much memory out there so much ability to "absorb" and retransmit. That gives the network a sort of 'resonant' frequency with respect to packet retransmission. I could imagine doing the Tesla trick of injecting burst of packets into the network at the resonant frequency and being able to get all the buffers to explode.

simcop2387• 14 years ago

All that being said, there is still something to be considered about adjusting the parameters to more closely match the network as it is today. That's mostly what this is going on about, not about completely doing away with congestion control, but instead making it fit better with the higher bandwidth connections that exist. What the right changes would be I have no clue.

po• 14 years ago

The converse is that everyone starts out going as fast as they can

As far as I can see, nobody is recommending that. This isn't about getting rid of slow-start. They're just talking about tuning the initial window to be larger.

If you were writing this algorithm today, you would look at the value for the window that most systems end up on and then pick an initial value just slightly below or equal to that. I doubt that number would be 2.

etrain• 14 years ago

Q: do you think the designers of TCP were thinking of the constant that they started with in the first place? Or do you think as computer scientists they said "some integer n > 0 so that we can achieve the exponential effect."

While we're modifying the initial constant, why not reconsider the exponent? Or the shape of the equation itself?

smosher• 14 years ago

He was exaggerating a bit when he said "tens" of years ago. It wasn't that long ago that dialup was downright pervasive in North America and Europe. At that time it didn't take too much abuse to discover just what an overstuffed pipe would do to your connection.

On the other hand, we have so much more bandwidth now. 56kBd is a fart in today's gale. Of course we shouldn't start out at full throttle, but the factor of 5 he's endorsing is virtually nothing and it still gives a higher proportion of overhead than we had back then.

sams996• 14 years ago

This is very far off from what I am proposing, the Linux kernel team decided to implement a recommendation that is still under review by the ietf, every release of Linux from 3 onwards has IW 10 enabled by-default. Like it or not, IW of 10 is probably here to stay. The 3 line has 3 stable release branches, all with this change. The majority of web servers online are still running 2.

skyraider• 14 years ago

Hm - spreading out the ramping-up over time may not be necessary for getting the window up to size while keeping congestion low. One idea that I found intriguing is "probing" the appropriate window size by "trying out" a large range of window sizes in a single round-trip time: http://www.cs.unc.edu/~jasleen/papers/infocom09.pdf

So you can start slowly, but time isn't necessarily the axis along which your start is "slow" or "fast." I am really interested to see whether the above or a similar protocol will be able to spread out "slow"-start over an axis of window sizes that the protocol probes for. Then maybe we will be less concerned with who gets to decide what the initial congestion window is, because you can't ramp up too "fast" when you know what to ramp up to.

sams996• 14 years ago

I also came across this http://yuba.stanford.edu/~nanditad/talks.html which is very interesting

barrkel3• 14 years ago

I object to your metaphor; stop signs do slow down traffic. If every 4-way stop sign junction had a roundabout (either full-sized or mini, depending on available space), traffic wouldn't need to stop very often, overall throughput would be much higher, and accidents would probably be even less.

4-way stop sign junctions were probably the most asinine, time-wasting, fuel-wasting road control I found when I drove in the US.

vacri• 14 years ago

Discussing this with a US friend, I have come to the conclusion that for single-lane roads, a roundabout is superior to a 4-way stop as you only have to watch one direction for traffic.

Once you get to multiple lanes, the answer is simple: both roundabouts and 4-way stops are inferior...

epo• 14 years ago

Can't comprehend why you were downvoted. Unless Americans are aware of a more asinine, time-wasting, fuel-wasting road control.

sgt2• 14 years ago

We also have 4-way stops but you don't actually stop entirely, that would be counterproductive wouldn't it. You "stop" (in a grey sense of the word), look left and right, then you go.

protomyth2• 14 years ago

4-way stops are popular because of construction cost and space savings. Different optimizations yield different problem areas.

brlewis2• 14 years ago

INACCURATE TL;DR

I don't usually post in all caps here, but many readers, myself included, will sometimes go to HN comments first to get a TL;DR version of an article. The thoughtful rebuttals by po and sams99 can easily be overlooked by people in a hurry. I think it's too late to edit or delete the parent comment, and the "flag" function is for spam and uncivility, not for carelessness.

As po and sams99 pointed out, the article is not about not doing slow start. It's about tweaking slow start parameters. This particular tweak has been adopted by the Linux kernel authors and others who do understand why TCP slow start is important.

ChuckMcM4• 14 years ago

Yes, its about changing the slow start congestion window from 2 to 10. And the author accurately points out that this is a huge win for your basic web page.

But lets think about that from the other side for a moment. Most systems will end up with a congestion window size of 1452 because their router has an MTU of 1500. With a CW of 2 that is 3000 bytes blasted out waiting for an ack (even though its only 2904 data bytes the MTU constraint will fragment it into two 1500 byte packets), with a CW of 10 that is 15000 bytes blasted out initially.

Lets say everyone on Comcast's network gets this change, they have an estimated 15 - 20 million subscribers. We'll be conservative and call it 15 million, and at any given time maybe half of them are doing an HTTP request (think about all the things that do HTTP requests in your house for a moment). So at any given instant in time you've gone from blasting out 3000 * 7.5M or 22.5GBytes (or 225 Gigabits) to 112.5GB or 1.125 terabits of data. Not surprisingly a lot of their traffic goes to some peering network and now their hammer is hitting with Terabit whacks instead of 200 Gigabit whacks. That is a noticeable change, and its an even bigger change when you consider they are running VOIP traffic as well.

Some useful work to be done here is to look at where the congestion window settles out on your network. And for what servers. And for which transit networks. How often does it get to 10? 20? The worst case would be it settles at 16. That's because with an IW of 10 the next window is 20, and then you get clamped.

There has been a lot of great work done on congestion control, and yes its really annoying like metering lights[1] when its not needed. But when it is needed it makes the system work.

Congestion is a function of cross section bandwidth and traffic demands. The cross section bandwidth of the Internet has gone up a lot, the number of 'ports' on the Internet has gone up even more. The ratio, has not improved much (and in some cases gotten worse) from the days of dial up.

I fully recognize that a number of people have done the same test (or thought experiment) the author has done with slow start and seen a green field for improvement. I was reasonably active in the IETF before the work on congestion was implemented, with a protocol that could be very latency sensitive (NFS), and it was a hellish environment. My disagreement, is that changing slow start in this way will destroy a number of interesting streaming services by increasing the standard deviation on latency outside of what they can tolerate. And for what? So that a 34K web page loads 30% faster? I'd much rather compress the web page or build a web service protocol that knows about slow start and accommodates it than this.

As others have pointed out, this change is going to happen regardless, and perhaps I'll have the opportunity for an 'I told you so', or perhaps I'll be relegated into the dustbin of ranting network dudes from the last century. But I stand by my TL;DR that the author did not demonstrate an appreciation for the impact of changing slow-start would have on the network in general because they focused only on how it would make their life faster.

[1] Here in California we have congestion control 'metering' lights on some on-ramps to the freeway, its annoying as hell when they are on and there isn't anyone on the freeway.

snissn3• 14 years ago

If upgrading my Linux Kernel will solve all of my problems, why is the experimental comparison between a Linux box and a windows box? Just saying..

sams996• 14 years ago

mainly cause I did not have a chance to set up an old Linux VM. The number hold though, the initial congestion window is 2-3 on the 2 line kernels.

snissn3• 14 years ago

I don't disagree, but it makes it an apples an oranges comparison. It introduces the variables of how linux vs windows deals with TCP (not withstanding that linux 2.x vs 3. might have some internal ipv4 changes, but youre recommendation is to upgrade anyway, so that's fine) but also changes in the webserver.. It seems like the changes are hard coded into the compiled kernel, so there's no way to simply change configuration flags?

That said, thanks for the post, and I'll definitely be tcpdumping in the upcoming week and reading some more about slowstart!

Maybe testing with net.ipv4.tcp_slow_start_after_idle 0 vs 1 would make a cleaner comparison?

yorhel2• 14 years ago

Not really a solution for the short term, but CCNx[1] looks like it'd solve a lot of problems that TCP currently has for both large file transfers and short web browsing. 1. http://www.ccnx.org/

obtu• 14 years ago

That site is terribly vague, what are the specific problems and how does CCNx address them?

yorhel2• 14 years ago

It would indeed be nice if the project had a proper introductionary page or something. Either way, the paper "Networking Named Content" is pretty much the best introduction you can get, and a very interesting read at the same time. Googling gave me a PDF at the following URL: http://conferences.sigcomm.org/co-next/2009/papers/Jacobson....

alpb2• 14 years ago

I wonder if Apple has implemented this in OS X kernel.

Flow• 14 years ago

sudo sysctl -w net.inet.tcp.slowstart_flightsize=10

alpb2• 14 years ago

Wow it changed from 1 to 10. Should I put that to boot script or what do you recommend?

lloeki• 14 years ago

This article gives the IW status on Windows and Linux. What is the status on other systems (e.g Mac OS X, FreeBSD, Solaris...) ?

Does it matter only server side, or do clients benefit of having this window increased too?

Also, a comment of the article mentions this:

    Why are you talking about upgrading the kernel, when you can simply do:
    
        ip route change default via MYGATEWAY dev MYDEVICE initcwnd 10

which would be similar to the netsh tunable on Windows. So upgrading the kernel is only needed to have it set to 10 by default.

EDIT:

It seems Mac OS X is using either NewReno or LEDBAT instead of the mentioned CUBIC or Vegas. Look for tcp_ledbat_cwnd_init in [1] which looks quite simple, or tcp_newreno_cwnd_init_or_reset in [0] which looks a bit more involved:

    /* Calculate initial cwnd according to RFC3390,
     * - On a standard link, this will result in a higher cwnd
     * and improve initial transfer rate.
     * - Keep the old ss_fltsz sysctl for ABI compabitility issues.
     * but it will be overriden if tcp_do_rfc3390 sysctl is set.
     */

PS: xnu-1699.24.23 is Lion 10.7.3

[0] http://opensource.apple.com/source/xnu/xnu-1699.24.23/bsd/ne...

[1] http://opensource.apple.com/source/xnu/xnu-1699.24.23/bsd/ne...

gizzlon• 14 years ago

This Freebsd related paper[1] is a little old but seems to suggest: "it depends". I didn't actually read it..

Tuning and Testing the FreeBSD 6 TCP Stack (2007) http://caia.swin.edu.au/reports/070717B/CAIA-TR-070717B.pdf

sams996• 14 years ago

re the ip route trick, be sure to read http://lkml.indiana.edu/hypermail/linux/kernel/1007.2/00395....

The_Fox• 14 years ago

The initcwnd change is helpful on any host that has more than 2 segments worth of data ready to send at the beginning of the connection. So a client that wants to send lots of data would benefit from the change.

For 99% of web browsing, the client's request fits in one or two segments and so would not benefit from the change.

sgt2• 14 years ago

In OS X all you need to do is set a sysctl setting. No need for even a restart. Check out net.inet.tcp.slowstart_flightsize (set it to 10) if you are interested.

ilaksh• 14 years ago

What versions of Ubuntu have this larger (faster) setting?

mixmastamyk• 14 years ago

Looks like Oneiric and Precise have > 3.0 kernels.

jcastro• 14 years ago

And the Oneiric kernel is backported to 10.04LTS:

Installing "linux-image-generic-lts-backport-oneiric" outta do it.

pkh80• 14 years ago

I'm sure there is a chance I am missing some bits here, but I setup a 3.0.18 / Ubuntu 11.10 / Apache 2.2 server and to compare against 2.6.39 Apache 2.2 server and in all tests they are basically identical.

sams996• 14 years ago

yeah ... the change was introduced in 2.6.39 ... but its a pretty rare kernel to have afaik, not even in debian backports anymore

Drbble• 14 years ago

Linkbait title. Editors, please fix.

Why upgrading your Linux Kernel will make your customers much happier

Related Stories