A Higgs-Bugson in the Linux Kernel

https://blog.janestreet.com/a-higgs-bugson-in-the-linux-kernel/

by Ne02ptzero • 1 year ago

232 55 1 year ago

blog.janestreet.com

protocolture• 1 year ago

I love the term "Higgs Bugson". Its much better than what I usually do which is just call a system haunted.

EarlKing• 1 year ago

Haunted? Hell, it's positively possessed.

GTP• 1 year ago

I was used to the more common Heisenbug, but I find Higgs-Bugson more funny.

burnt-resistor5• 1 year ago

There is no magic at any time. All behavior lives somewhere.

hglee• 1 year ago

https://lists.openwall.net/linux-kernel/2025/03/19/1374

penguin_booze• 1 year ago

I wish developers--new and old alike--pay attention to the commit messages that goes into the kernel. Granted, it takes a subject matter expert to really understand what's being said, but the general format and layout of commit messages is instructive. Commit messages helps the reader/reviewer get their bearings; they also help to build the case from the bottom up.

The fact that the development team is globally distributed both necessitates this kind of knowledge serialization and preserves it for posterity. It's completely different from tapping a colleague sitting next you on the shoulder, and saying "psst, can you approve this quick? It's just a bunch of fixes".

andrybak2• 1 year ago

The kernel repository has pretty extensive documentation on how to describe the code changes in the commit messages:

- https://docs.kernel.org/process/submitting-patches.html#desc...

- https://github.com/torvalds/linux/blob/master/Documentation/...

- https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Git's own guidelines also have a nice description on how to write a better commit message:

- https://git-scm.com/docs/SubmittingPatches

- https://github.com/git/git/blob/master/Documentation/Submitt...

- https://git.kernel.org/pub/scm/git/git.git/tree/Documentatio...

andrybak2• 1 year ago

A different UI to the mailing list with the whole thread on one page and diff coloring, if anyone is interested: https://lore.kernel.org/lkml/20250319-rfc2203-seqnum-cache-v...

sedatk• 1 year ago

A higgs-bugson is a bug that is reported in practice but difficult to reproduce

This was the first time I heard of "higgs-bugson". The term sounded so forced that I had to know how it differed from Heisenbug. In short, it doesn't[1].

Then why did it even exist?

The term somehow made it to the "Heisenbug"'s Wikipedia page[1], so I checked the sources. There were two and both end up at the same site: Jeff Atwood's blog post[2] quoting some StackOverflow answers to a poll-like question ("what's a programming term you coined?") because he wanted to remove lighthearted content from the site as he thought it clashed with SO's mission of educating people and advancing their skills[3].

There was a proposal on Meta StackExchange about undeleting that question with the answers, but it was refused by Jeff Atwood again because it invited "made up stuff"[4] among other reasons.

So, Wikipedia in the end, has this term in Heisenbug page because someone just blurted out something in 2010, it was copy-pasted to a blog, and then got scooped up by some news outlet. There are no other sources. Kagi doesn't find any instances of the term before it was coined on StackOverflow in 2010. For all we know, "gingerbreadboy" from England invented it.

The irony is that the term somehow made it to the literature -hence the blog post here- because someone was just having fun at StackOverflow. It obviously either sounded good, or just clicked that others started using it. StackOverflow deleted the content that actually made a small part of computer science history because it wasn't "serious".

In other words, StackOverflow cut off one of its history-making parts because it had an incomplete and simplistic view of useful. I think it might be possible to draw a line from their understanding of communities and societal dynamics to the downfall of StackOverflow after the emergence of AI[5].

[1] https://en.wikipedia.org/wiki/Heisenbug

[2] https://blog.codinghorror.com/new-programming-jargon/

[3] https://stackoverflow.blog/2010/01/04/stack-overflow-where-w...

[4] https://meta.stackexchange.com/questions/122164/can-we-un-de...

[5] https://blog.pragmaticengineer.com/stack-overflow-is-almost-...

chris_wot• 1 year ago

Yeah, stack overflow is dying, we all know it.

zahlman• 1 year ago

because he wanted to remove lighthearted content from the site as he thought it clashed with SO's mission of educating people and advancing their skills[3].

No; he wanted to remove discussion and socialization, because it clashed with SO's mission of presenting useful information without parsing through others' discussion.

https://meta.stackexchange.com/questions/2950

https://meta.stackexchange.com/questions/19665

https://meta.stackexchange.com/questions/92107

https://meta.stackexchange.com/questions/131009

In other words, StackOverflow cut off one of its history-making parts because it had an incomplete and simplistic view of useful.

How does this in any way demonstrate that the view of usefulness was "incomplete" or "simplistic"?

How is the deleted content "useful"?

I think it might be possible to draw a line from their understanding of communities and societal dynamics to the downfall of StackOverflow after the emergence of AI[5].

What downfall?

Before you point at any of the incoming-question-rate statistics: why should they be interpreted as representing a "downfall"? That is, why is it actually bad if fewer questions are asked?

Before you answer that, keep in mind that Stack Overflow already has more than three times as many publicly visible questions about programming as Wikipedia has articles about literally anything notable.

robertlagrant• 1 year ago

why should they be interpreted as representing a "downfall"?

I agree, but also SO has certainly gone through ups and downs. It does feel as though it's now in a terminal "down" having invested its limited resources in things lots of the dedicated members didn't seem to want, instead of basic improvements to moderation and to chat features.

dh2022• 1 year ago

I think Heisenbug refers to a bug that stops repro’ing during debugging (the act of observing the system changes the system behavior). This bug was different: it was very rare and debugging it didn’t make it go away.

bux93• 1 year ago

Like they say, "Stop trying to make fetch happen!"

anonymousiam• 1 year ago

"The normal timeout logic can take care of retransmission in the unlikely case that one is needed."

NFS can be run over TCP or UDP. Does the retransmission occur when using UDP?

ninjha• 1 year ago

Yes! The retransmission logic in Linux NFS is independent of transport (see the `retrans` option in `mount.nfs`).

Weirdly enough this also means that if you’re running with TCP you can have retransmits at the NFS/SunRPC level and at the TCP level depending on your configuration.

konsalexee• 1 year ago

TIL higgs-bugson and Heisenbug

snvzz3• 1 year ago

With millions of LoCs, it is no surprise there are bugs.

Worse yet, the kernel runs in supervisor mode.

This kernel design is bankrupt. There's much better available, such as seL4+Genode.

eddythompson80• 1 year ago

seL4+Genode is equally as bankrupt. I run my code in the SMM anyway.

eqvinox7• 1 year ago

Please try keeping your snide comments to issues they actually apply to. This is a logic bug, with the kernel missing a piece of abnormality handling. You can get the exact same bug in a microkernel (or, FWIW, a memory safe, e.g. Rust) implementation; neither of those concepts help here.

snvzz3• 1 year ago

You can get the exact same bug in a microkernel

Absolutely. And yet, it is that much easier to keep a tiny codebase bug-free.

And only that tiny codebase has to run with supervisor privileges.

lotharcable• 1 year ago

This kernel design is bankrupt. There's much better available, such as seL4+Genode.

I am sure that the tech community would love to read the details of your great success in deploying microkernels for large variety of production workloads.

burnt-resistor5• 1 year ago

seL4 exhibited great advances in software engineering processes and advances in correctness, zero-copy microkernel IPC performance, and capabilities-based security, but these need explanation, adaptation, and evangelism to real-world use-cases like Linux.

Microkernels have severe limitations when it comes to transactional boundaries of calling multiple subsystems and rolling back on failure.

Linux has too much inertia to reinvent itself instantly or completely into XYZ.

What would add more value would be gradual conversion to Rust and adding formal verification to C and Rust like specifying invariants in comments/metadata like frama-c and/or flux.

PS: Religious judgement opinion wars are rarely constructive.

Havoc• 1 year ago

Didn't know jane street did tech writeups

Balinares• 1 year ago

It's a great recruitment device. It takes a certain kind of nerd to salivate over the glorious technical depths that such a write-up goes into, and for the kind of company who values this flavor of nerd, this is a great way to attract their attention.

burnt-resistor5• 1 year ago

OCaml and OxCaml https://blog.janestreet.com/introducing-oxcaml/

vrnvu2• 1 year ago

I'd like to highlight this:

NFS with Kerberos

secure, simple, battle tested. no crazy architecture

works so well a bug showed up in the kernel :-)

eqvinox7• 1 year ago

works so well a bug showed up in the kernel :-)

What exactly are you trying to highlight here? Most code has bugs. This one is someone forgetting to stick to actual behavior described in 1997, it's a mistake, mistakes happen. Which one of "secure", "simple", "battle tested" and "no crazy architecture" do you think this disproves?

Or do you think CIFS or Ceph have no bugs?

gyesxnuibh• 1 year ago

I think they're saying typically the kernel one of the last places you'd expect the bug, so it shows that it is battle tested?

I don't think they're being snarky.

burnt-resistor5• 1 year ago

The Linux kernel needs to adopt better testing methodologies because they're almost entirely reliant on meatcloud CI than provably-correct code with invariant contracts.

gnfargbl• 1 year ago

Calling this a "Higgs-Bugson" doesn't make a lot of sense. There's nothing uncertain or difficult to reproduce about the Higgs.

The reason that it took so long to find was that the cross-section of production is very low, the decay signatures are hard to separate from the background, the specific energy scale it existed at was not well-defined, and building the LHC was (to put it mildly) difficult and expensive.

Roughly, if you'll forgive a bad analogy from a long-lapsed physicist, it was the equivalent of trying to find a very weak glow from a specific type of bug hiding at an unknown location in a huge field of corn. Except that your vision was very bad, so you had to invent a new type of previously-unimaginably excellent eyeglasses to see the thing. Also before you could even start looking you had to expend a painful amount of time and money building a flashlight so incredibly huge that it needed new types of cryogenic cooling inventing, just to stop it from melting when you switched it on.

If you had a software bug that you were almost certain was there, but you needed half of the world's GPU clusters for three years to locate and prove it, then that would be a Higgs-Bugson.

lisper• 1 year ago

A bug that shows up in production but goes away when you try to debug it is usually called a Heisenbug.

Agingcoder• 1 year ago

Yes - happened to me once. I could reproduce it consistently until I turned tcpdump on , at which point it would go away. I resorted to ebpf as well.

alexpotato• 1 year ago

Regarding NFS, I've always loved this quote from the CTO at a hedge fund I once worked at:

"NFS is lot like heroin:

at first, it seems amazing.

But then it ruins your life"

(This is a place that did almost EVERYTHING via NFS including different applications communicating via shared files on NFS mounts. It also had the weird setup of using BOTH Linux AND Windows permissions on NFS mounts shared between user desktops [windows] an servers [linux])

stavros• 1 year ago

The problem I have with reviews like these is that they're expressed in absolute terms. Yes, NFS might ruin my life, but if it ruins my life less than every other alternative, it's still a win.

eqvinox7• 1 year ago

I'd go as far as saying most networked concurrent file access will ruin your life one way or another, because it's just a hard problem, and it's trying to solve it at a very odd layer; a "classic" fs can't really take advantage of higher layer transactional or other known constraints in order to make things work better…

johncolanduoni• 1 year ago

Continuing the analogy, many people eventually discover that they used NFS because they didn’t understand their underlying problem clearly.

burnt-resistor5• 1 year ago

The devil is in the details, the devil you know is preferable, and there's yet no perfectly angelic systems or code (because of the widespread allergy to formal methods and job security).. which will lead to less evil, but still imperfect systems.

jxjnskkzxxhx• 1 year ago

Content marketing for Jane street.

mikeyg• 1 year ago

anyone else feel like the linux kernel release quality has come down a bit here in the 2020s? i feel like it hasn't been this bad since the mid 90s. anecdotally in the past couple years, i've experienced a data corruption bug in xfs, wonky wifi firmware/kernel regressions, graphics artifacts and hard crashes in amdgpu. my experience with mainline releases before 2020 has been that they're rock solid. i'd doubt myself before i doubted the kernel. i say all this with a deep appreciation for everyone and the work they're doing... my intuition says that the complexity of it all is reaching a tipping point that is finally overwhelming the ages old release engineering processes.

jcalvinowens• 1 year ago

My experience is the opposite. I run mainline starting with -rc1 on my gaming pc, its been literally flawless for me since I switched to an AMD RX 7900 XT about two years ago. Ten years ago there was a 50/50 chance -rc1 would fail to boot on at least one of my machines, I can't remember the last time that happened.

Look at all the syzbot automation on the kernel mailing list, as an example of how the process continues to improve.

IMHO the best Linux experience really is to run bleeding edge versions of everything. I wasted a great deal more time backporting patches before I started doing that, than I have spent chasing new bugs since.

nycerrrrrrrrrr• 1 year ago

Curious why they're using NFSv3 instead of v4?

emmelaich• 1 year ago

Kerberos is 'fun'. I had to manage a system which used Kerberos to provide authentication between a Rubik's cube of various Windows flavours with various crypto standards, Linux machines of various versions and Java versions and apps of various maturity. It was an ever present source of weird behaviour and I had to bury myself in the innards of all these systems.

I know, not directly related to the article. Just needed to vent bitterly.

alienbaby• 1 year ago

I always thought this was rather called a 'Heisenbug'?

12_throw_away• 1 year ago

I think the distinction the author is making is that:

- a Heisenbug is stochastic and potentially non-local, but

- a Higgs-Bugson is a bug that is known to exist (in prod) but is extremely hard to observe in the lab (during dev/testing).

I can see it being a useful distinction. A bug that I can't even reproduce at all sits in a different ring of hell from, say, a memory corruption bug that makes the program crash on random unrelated lines of code.

rurban• 1 year ago

NFS is usually only used in mixed linux/windows environments. The easiest fix is to avoid NFS and esp. Windows. NFS alone is nightmare enough, Windows is just insanity.

saagarjha• 1 year ago

I think most of the places I've seen NFS used are all UNIX/Linux shops. Why do you think Windows is commonly involved?

jwillp• 1 year ago

The article is well-written and I even learned a few things. I'm glad for Nikhil's persistence troubleshooting it and fixing the bug upstream. Thanks, Nikhil!

A Higgs-Bugson in the Linux Kernel

Related Stories