Google fixes nearly decade-old Linux kernel TCP bug

http://bitsup.blogspot.com/2015/09/thanks-google-tcp-team-for-open-source.html

by tim_sw • 11 years ago

355 46 11 years ago

bitsup.blogspot.com

Remember that TCP is robust enough that it seems to work anyhow - even at the cost of reduced network throughput in this case.

That to me sounds like this is not really a bug, but a performance enhancement; a bug would be something like corrupted/missing/extra data at the application level. Tuning the TCP algorithms beyond the basics is really more of an art.

wfunction• 11 years ago

It's a bug if you consider performance to be a feature.

plorg• 11 years ago

Presumably it is considered to be a bug because it does not accomplish what the code is meant to accomplish - namely, the code does not properly characterize a congested state and thus does not operate as if it were congested.

nathanb2• 11 years ago

Given your definition, and viewed from a high enough level, every feature or enhancement is a bug fix. I think nitpicking the wording (while an interesting philosophical discussion) detracts from the point of the article: an unnamed Google engineer waded into the Linux kernel code and made the web slightly better for many people.

smegel2• 11 years ago

I thought that too, but then

that an endpoint that is not moving any traffic cannot use the lack of errors as information in its feedback loop.

I think the bug is that it was counting something when it should have not been, which would have violated both the design and intent even at the time it was written.

But I agree that code is only a bug if it does not do what was originally intended, if the original design was bad, and the code correctly implemented that bad design, there is no bug.

sigmaml2• 11 years ago

But I agree that code is only a bug if it does not do what was originally intended, if the original design was bad, and the code correctly implemented that bad design, there is no bug.

The frame of reference for determining if something is a bug or not is the requirement specification, not design. If the design incorrectly or inadequately addresses the requirement, that is a bug as well. There isn't much point in rejoicing over the code that correctly implements an incorrect design!

kpil• 11 years ago

Programming is 100% design, there really is no separate "implementation" phase.

If the design has a flaw, whatever the reason: it is a bug.

A big part of development is "discovering the requirements." It is typically the hardest part for real world projects, and when that fails, it is perhaps not a bug in the code, but it still a development error that needs to be fixed.

If a developer got a piece of paper that said "blue" but the real world requirement was "green", and it was not possible for the developer to find out together with any stakeholder, then there is a development process error, that needs to be fixed.

In any event, it needs to be fixed, so pointing to someone else wont really help.

rasz_pl• 11 years ago

not really a bug

Imagine a car turning off traction control in a pouring rain every time you stop at the lights because 'Hey,no tire spin in the last 5 seconds!!1'

simoncion• 11 years ago

I don't see why this would be a problem. Who on earth would jackrabbit from a stop in the pouring rain?

gozo• 11 years ago

I don't think "bug" is a very well-defined term. It seemingly encompasses most software imperfections. I guess people will generally characterize things based on their overall (negative) impact.

thecaviardoer• 11 years ago

Honest question: How is it that Google is the company that's always doing all of this involved technical work? Or is it just that their work gets publicity?

adventured• 11 years ago

They have tens of thousands of engineers and work with vast amounts of open source software. Only a few other companies are of comparable scale in terms of sheer software engineering. I think it's inevitable that they'd have a very large footprint across the industry.

vezzy-fnord• 11 years ago

At least in the case of the Linux kernel, Google is actually around the 9th or 10th largest corporate contributor. [1] I'm not sure if they deserve to be particularly specially exalted, given the vast body of kernel work unrelated to them.

[1] http://www.linuxfoundation.org/publications/linux-foundation...

tga_d• 11 years ago

Looking at the list here: [1] Most of the top employers are hardware companies, and therefore likely to contribute mostly driver patches. For example, the rankings by LoC for this release list AMD as over a third of the contributions, almost certainly because of their new AMDGPU driver. Drivers are important, of course, but hardly news-worthy. Red Hat is always near the top of the list, and they do a lot of both the drudge work and the interesting stuff, but I feel as though everyone in the Linux community already knows that.

[1] https://lwn.net/Articles/654633/

Tobu• 11 years ago

Éric Dumazet (author of the patch) is this highly productive guy working on network latency issues. He helped get CoDel into the kernel, and worked on TCP small queues (https://lwn.net/Articles/507065/) and the fair queue scheduler (https://lwn.net/Articles/564978/). Looking at the email addresses he used, I think he started working on the network stack on his spare time, while employed at SFR, a French telco. He implemented an in-kernel JIT for BPF (https://lwn.net/Articles/437981/), speeding up packet filtering; now that BPF is fast it is also used for performance profiling (filtering and aggregating in kernel) and syscall filtering.

There are of course others working in these areas, but I think Google has been going after them: Van Jacobson is a high-profile example. Dave Taht is still subsiding on ramen though (https://www.patreon.com/dtaht).

sandGorgon• 11 years ago

For the last five years I've worked myself to the bone, mostly unpaid, to solve the "bufferbloat" problem - first by organizing and leading the team that solved it, and giving away, for free, the ideas and code for anyone to use! And I've spent tons of time later convincing standards organizations like the IETF to make the ideas standard in new equipment - and also open source makers like openwrt to retrofit the fq_codel and cake fixes into millions of older yet upgradable home routers, and helping ISPs, vendors and chipmakers understand the both the need for the fq_codel fixes, and how to implement them.

wow - thanks for posting about David Taht. someone like Netflix or Google should get him on their payroll.

mackal• 11 years ago

They have the money the hire those that can fix the issues. The issues also greatly affects their business, so they allocate resources to look into them. Less latency in the network stack means faster response time, means more ads delivered in less time. And a smoother user experience.

eridius• 11 years ago

I'm confused by the last sentence:

The whole web, including Firefox users, will benefit.

Why is Firefox thrown in here? And the post is even tagged with "firefox". But the contents of the post seems to have absolutely nothing to do with Firefox, except inasmuch as Firefox is an application that interacts with the web.

TimothyFitz• 11 years ago

The author works for mozilla, and firefox is a common theme on his blog. I think it's for regular readers of his blog to understand the practical impact (readers he assumes care more than average about firefox).

stuaxo• 11 years ago

I'd guess that because it was a bugfix by google - people might assume it was something to do with chrome or android, when it should help any webuser that talks to a linux based server, including firefox users.

Manishearth• 11 years ago

The tag is probably so that the post will show up in Planet Mozilla (http://planet.mozilla.org).

I often tag non-Mozilla posts as "mozilla" if they are still relevant to Planet's scope on my blog too. I've submitted a category feed instead of the full RSS blog feed so that I can exclude things which are physicsy or otherwise irrelevant.

nathanb2• 11 years ago

I would love to see a characterization of this bug's impact. Reading the description, it doesn't seem to be nearly as serious as the author of this post is making it out to be. But there's no information about how fast the client will step up the transmit rate after a period of quiescence and no data from traffic analysis to determine how frequently real-world servers encounter this condition and the overall effect in terms of dropped packets, increased queue depths, and latency spikes.

(Google are actually in a good position to provide these data; a number of Google's own services will doubtless benefit. I suspect the scale of Google's operations means that even a barely-measurable decrease in latency or queue depth compounds to a big gain pretty quickly.)

cornholio• 11 years ago

Strictly speaking, it can't be too bad, since it would immediately stand out if Linux boxes have decreased network performance in a mixed shop. Maybe a few percent in the average case, with maybe some specialized applications more affected.

toolslive• 11 years ago

There is a talk https://www.youtube.com/watch?v=gQsOo_skjzk Where they show there are plenty of similar bugs left in TCP implementations. It's a mess.

mtkd• 11 years ago

But less of a mess today than it was 17 days ago.

Supersaiyan_IV2• 11 years ago

The reason this is good is because older wifi cards supported by iwlwifi get hardware buffer underruns and hangs when said bug occurs together with WPA2 enabled.

tjoff2• 11 years ago

I'd say that is a reason for why this is bad. If I've got bad hardware I want to know about it and bad hardware should be flushed from the market. Not function "okaish" enough to not annoy users to the point of boycott.

Supersaiyan_IV2• 11 years ago

Allow me to clarify. "This" means: "The fact that Google fixed a decade old bug", meaning the topic itself. The "this" you use in your comment is entirely different. The above misuse of "this" causes me to be confused, because it allows your comment to be interpreted as if you're promoting the inference of bugs, where older hardware is affected, such that people lose faith in that hardware faster due to frustration, consequently affecting the market causing it relieve itself of older hardware faster. Which is why I'm asking you to clarify your argument.

zurn• 11 years ago

The commit also says "This particularly shows up when slow_start_after_idle is disabled as a dangerous cwnd inflation (1.5 x RTT) after few seconds of idle time." And the slow_start_after_idle sysctl is enabled by default.

The bug fix is in the CUBIC congestion control algorithm, which has been the default in Linux TCP since kernel 2.6.19 (released in November 2007). So impact in distros has been ~ 2008-2015.

zaroth• 11 years ago

Speaking of Linux kernel network stack enhancements, there's one I'm interested in but not sure where to start.

I was trying to setup ECMP on Ubuntu 14 and found I was constantly getting TCP RST. Turns out after digging into the code I found the path selection algorithm is not a consistent hash of a packet header tuple but rather pseudo-randomly chosen.

In older kernels there was a route cache so a path would be chosen and cached and so you would get a stable ECMP route per source. The route cache was removed so now the random selection runs on every packet leading to unstable routing.

It seems a simple fix of choosing the right hash function and maybe adding some configuration flags to determine which fields are hashes on (typically 5-tuple SrcIP, DstIP, Ip Protocol, TCP Src Port, TCP Dst Port).

But what would be incredibly cool is if my patch could one day possibly even merge to mainline. Is it reasonable to just blast a 'here's what I'm trying' out to netdev mailing list to get some feedback? I must admit I'm a bit intimidated to post...

jfoks• 11 years ago

Absolutely go for it! Post it even if you feel uncertain about it, just be clear about that, and about what your patch is trying to achieve, when the unexpected behaviour happens, and try to include a way for others to reproduce/investigate (as simple as possible, perhaps a program that demonstrates the bug). Also be clear about how you feel about whether or not the patch is the right approach and about what you would like list members to do with it (are you looking to confirm that what you're seeing is a real kernel bug, or a misunderstanding, or an application bug, and/or are you looking for help solving the bug/issue you're seeing, and/or are you looking to get it merged, etc). Can you demonstrate/quantify how your patch improves/fixes things, etc.

Obi_Juan_Kenobi• 11 years ago

Open source is generally quite kind to newbies, provided you provide that important context. Try to explain not just what you're doing, but also be very clear about why. Explain the problem as well as you can, and then discuss what solutions you've considered thus far.

It sounds like you have a good understanding of the problem. If anything, I'd look into why the route cache was removed; perhaps there was a good reason for this? If nothing else, you can ask and learn something.

Google fixes nearly decade-old Linux kernel TCP bug

Related Stories