Who Writes the Bugs? A Deeper Look at 125,000 Kernel Vulnerabilities

https://pebblebed.com/blog/kernel-bugs-part2

by MBCook • 5 months ago

71 23 5 months ago

dogleash• 5 months ago

These smell like the kind of metrics that cause someone to feel informed and then to miss the forest for the trees. The kind of data for a "data driven" decision maker who will just invent a narrative to explain the numbers, and then do what they wanted to do all along.

The map is not the territory.

palmotea• 5 months ago

These smell like the kind of metrics that cause someone to feel informed and then to miss the forest for the trees. The kind of data for a "data driven" decision maker who will just invent a narrative to explain the numbers, and then do what they wanted to do all along.

We need to increase reliability in the kernel, so the kernel team should fire the top 5 bug-introducers, to reduce the amount of bugs being introduced (https://pebblebed.com/blog/kernel-bugs-part2/05_author_analy...). Linus has got to go.

gchamonlive• 5 months ago

We need to increase reliability in the kernel, so the kernel team should fire the top 5 bug-introducers, to reduce the amount of bugs being introduced (https://pebblebed.com/blog/kernel-bugs-part2/05_author_analy...). Linus has got to go.

You've cut bugs being introduced while also reducing development costs by slashing team size. You deserve a promotion and an increase in equity.

alwa• 5 months ago

The LLM-tone doesn’t help:

117 people meet this criteria. And the impact is dramatic:

It’s strange to me to think of “bugfixes” in terms of a commodity. Different problem spaces between subsystems and thus different types of (and surfaces for) bugs; different contributor mixes; different number of eyes on them; different potential impacts…

CAN bus drivers top the list [of bug lifetime by subsystem]. These are used in automotive and industrial systems. Critical infrastructure with few maintainers watching.

…or maybe higher-quality initial submissions, with most of the easy bugs already wrung out of them, so only subtle bugs remain (thus fewer to fix).

Or adequately vigilant maintainers but low diversity of systems running that code, thus fewer users/situations where the bugs manifest, so they go unreported. Or poorer telemetry so an ordinary rate of latent bugs but they go undetected.

Could be any, probably a little of all, can’t really tell from the analysis; and each cause would suggest a different response to improve quality.

gbacon• 5 months ago

I’d expect an LLM to use correct grammatical number in the first sentence that you quoted. As written, the demonstrative ‘this’ has singular number but ‘criteria’ plural.

jeffbee• 5 months ago

Bugs Georg, who is an outlier and should be excluded from the analysis.

olivia-banks• 5 months ago

Strange how someone in a cave with no internet can push 10,000 bugs a day.

petterroea• 5 months ago

Not happy with the lack of statistical testing, some of the smaller differences in % could probably be coincidence

vintagedave• 5 months ago

This reads like Claude wrote it (more than ChatGPT.) Interesting data but I am unsure how actionable it is. Are they suggesting, for example, that specific commit messages get scanner more closely? Why is CAN more severe than Intel? (It does worry me. I feel like bugs, of any sort, in car systems are terrifying.)

aDyslecticCrow2• 5 months ago

CAN more severe than Intel

I suspect the usage of the CAN driver in Linux is pretty low. The largest user of the Linux can driver is likely testing and diagnostics tooling for developing cars rather than the car themselves. Even when the car has a Linux computer, they often use multi CPU SOC's that run the real-time CAN logic separate from Linux, and only convey application logic into Linux.

I could also speculate that the overlap between Linux kernel developers and automotive and industrial embedded systems is pretty low. So the high bug severity in the CAN driver could be developers contributing patches from a very different programming background?

myself248• 5 months ago

I could also speculate that the overlap between Linux kernel developers and automotive and industrial embedded systems is pretty low.

Agreed.

So the high bug severity in the CAN driver could be developers contributing patches from a very different programming background?

Background and situation. Their mindset is "I need this to work, right now", not "I need this to work, and not break anything else, forever".

earthscienceman• 5 months ago

There are so many more embedded CAN systems beyond cars. Industrial battery management uses Linux and canbus, for example.

kittikitti• 5 months ago

I'm not sure why this isn't included in the blog, but I was curious about the ratio between bugs and commits. Presented here are my calculations in order of total number of bugs:

Intel : 11.86%

[1] Independent : 2.27%

Red Hat : 9.74%

Linaro : 12.73%

Google : 12.78%

AMD : 9.70%

The above is based on the bug count table in the article.

[1] I combined the total bug count for independent and kernel.org because they are combined for the total contributions here, https://github.com/quguanni/kernel-archaeology/blob/main/scr...

This suggests that corporations are introducing significantly more bugs than independent developers. However, I have not done statistical testing on this nor have I recreated the numbers. If I had to speculate, I would assume that the analysis from the author was partly vibe-coded or they purposely left this analysis out due to fear of retaliation. Extending my speculation would also include that corporations are purposely introducing bugs out of malice such that there are backdoors available for them. The author mentions that there is no "corporate takeover" but perhaps there are more interesting conclusions to be found.

rnewme• 5 months ago

What about the complexity and type of the committed code?

pixl97• 5 months ago

Ya, a total guess on my part is the corporations are adding more things like complete drivers and kernel modules where individuals may be adding more smaller fixes.

PowerElectronix• 5 months ago

You can feel IC fear of being roasted by linus in those numbers.

nickpsecurity• 5 months ago

Putting in a GPU or virtualization system, like Intel's, is probably much harder than the average, independent commit.

charcircuit• 5 months ago

I'd also like to see this broken down for C vs Rust.

vlovich1232• 5 months ago

Half the kernel is still built by individuals: people using gmail.com, personal domains, or university emails. The "corporate takeover" narrative is overstated. Companies contribute heavily, but the kernel remains a genuinely collaborative project.

Isn't the assumption here flawed? Someone may be employed by a corporation but still use their gmail/personal domain/university domain. This needs to be cross-correlated against some secondary source of employment data to give a more accurate picture.

andyferris• 5 months ago

I am a bit confused though here - should the kernel be written by unemployed people? Most contributors won’t have full-time funding.

fluoridation• 5 months ago

That's an assumption in the exact opposite direction. GP is assuming that if someone commits while employed by a company then that company paid completely for that commit, while you're assuming that in that case the company "probably" didn't pay for the whole commit.

Either way, the article's conclusion seems to be insufficiently supported.

Who Writes the Bugs? A Deeper Look at 125,000 Kernel Vulnerabilities

Related Stories