Response to systemd's use of the debug flag

https://plus.google.com/111049168280159033135/posts/Kd57G8s1cTD

by asimov42 • 12 years ago

38 32 12 years ago

plus.google.com

anon1385• 12 years ago

Correct. I don't mind people piggy-backing on some fairly obvious generic term like "debug" per se. I don't know if the old init scripts did that, but I do know they did it for "quiet", which is basically the reverse of "debug".
What I mind is people closing bugs and not admitting mistakes. If Kay had even said "sorry, the excessive output was a bug in systemd, it's already fixed in current -git", that would have been a valid reason to close the bug.

So Torvalds is saying he doesn't like dictatorial project maintainers who reply in an abrupt and abrasive manner to contributors? He is so concerned about the lack of politeness and professional discourse that he just had to raise this issue? Hilarious.

din-9• 12 years ago

> What I mind is people closing bugs and not admitting mistakes.
So Torvalds is saying he doesn't like dictatorial project maintainers who reply in an abrupt and abrasive manner to contributors?

You have quoted what he did say and made up a position that he did not.

azernik• 12 years ago

No. He doesn't like project maintainers who reply in an abrupt and dismissive way to users with bugs. In this, Linus is very consistent and not hypocritical at all.

rdtsc• 12 years ago

Hmm, I was expecting "Sorry we made a mistake, we'll switch the option to systemd.debug".

At that point there is simply no other option for that, because persistent storage is not available

This was about overloading an already used option by another team building a core system component -- the kernel. A debug for kernel's command line is for the kernel.

It's the option an admin can specify which tells him why the system doesnt boot,

Ok so he does and now his system also doesn't boot but now it is either because of the original problem or because it gets flooded by systemd logs.

And then, he goes and posts to the kernel mailing lists saying how kernel is a piece of shit.

That turns this into some kind of power game, which I am totally not interested in.

also

We are putting together an OS here after all, not just a kernel, and a kernel is just one component of the OS among many, and ultimately an implementation detail.

I think due to their attitude towards both testing, towards the kernel community, they shouldn't be building core system components. And did he just write that kernel is just "an implementation detail?".

Maybe systemd was a mistake. Integrating and dumping socket acceptors, logging, and the whole kitchen sink into one component. So when it breaks it really breaks.

qbrass• 12 years ago

And did he just write that kernel is just "an implementation detail?".

In theory, systemd could be made to work on BSDs, Hurd, Minix, or whatever.

In reality, I'd sooner expect systemd to provide it's own kernel, but you never know.

ZeroGravitas2• 12 years ago

The context for that remark is:

My system isn't booting. I don't know why. I'll pass "debug" to find out. Right here, and now, the fact is that the problem could be in many places inside or outside the kernel including (but not limited to) systemd. Requiring people to understand the current low level structure of their OS just to get some extra debug reporting on a failed boot is exposing implementation details.

Orangeair3• 12 years ago

I have a fair amount of experience with Linux, but I'm a little lost by what's going on here. Can someone out there dumb this down a bit for an init newbie?

lutusp• 12 years ago

In a nutshell, a Linux kernel developer wants to limit the number of debugging messages he is obliged to read and act on, and Linus Torvalds thinks that's counterproductive, and Linus generally says what he thinks.

In a nutshell. :)

lawl2• 12 years ago

You can pass the kernel options before booting. i.e. "quiet" or "debug". The Kernel exposes these options via /proc/cmdline.

systemd parses /proc/cmdline and if "debug" is set, it also raises it's log level to debug. Then systemd logs so much to the kernel log that it gets flooded and fails to boot.

Edit: And story wise, systemd says yeah, we're not going to fix this or rename the debug flag for systemd to "systemd.debug" because blah.

And Torvalds gets pissed.

Orangeair3• 12 years ago

So basically, systemd is catching the common keyword 'debug' from a command that being given to something else entirely, and in turn enabling its own logging, which floods the system. The command wasn't actually meant for systemd, but for the kernel. Is that correct?

lordgilman2• 12 years ago

The kernel has a debug option that can be specified at boot time to increase its logging verbosity. systemd looks at the kernel command line to do the same and starts writing its own verbose logging out to the kernel's message buffer. Writing out to the kernel message buffer is done for pragmatic reasons: generally if you're debugging init problems they're happening when the machine is first starting so you would like to see the error messages on the serial console/virtual console.

The bug that started this all came up when someone had a broken system that was causing systemd to spew out tons of log messages. This was overwhelming the kernel message system and causing things to time out/etc and resulted in an unbootable system. When this problem got opened upstream, the systemd maintainer said "not my problem."

There are a couple of ways forward here: systemd can limit its own output and say "my bad"[1], systemd can begin looking for its own debug flag in the kernel command line instead of using the kernel's own debug flag or the kernel can implement rate limiting on the message system[2] to keep userspace from shooting itself in the foot.

Torvalds really doesn't like the systemd maintainer's "not my problem" attitude so he's threatened to no longer accept pull requests from him for any code he writes. Torvalds wrote a really good post in the linked Google+ comments that describes his feelings on the whole thing:

"I don't mind people piggy-backing on some fairly obvious generic term like "debug" per se. I don't know if the old init scripts did that, but I do know they did it for "quiet", which is basically the reverse of "debug".

What I mind is people closing bugs and not admitting mistakes. If Kay had even said "sorry, the excessive output was a bug in systemd, it's already fixed in current -git", that would have been a valid reason to close the bug.

And for the people thinking this is a storm in a teacup: this is not the first time Kay has done this, which is why I personally get so frustrated.

Kay has done the exact same thing with major bugs that were not fixed anywhere else, and that caused machines to fail at boot time, and Kay happily pointed the finger elsewhere for months at a time and closed bugzilla entries.

Side explanatory note: and it's because of that known history of abusive behavior that I would prefer systemd now use "systemd.debug".

The old init scripts may or may not have parsed "debug", but we never had any reason to care. Now we do, and people are (I think) understandably upset that systemd not only screwed up, but then the people involved weren't even willing to say "sorry" about it but instead go "uh, it wasn't our bug, deal with it"."

----

[1] This happened

[2] This will likely happen

Orangeair3• 12 years ago

Very explanatory, thank you very much.

Tobu3• 12 years ago

systemd can limit its own output and say "my bad"[This happened]

Did it? I didn't find any such commit. IMHO it's the correct fix. /dev/kmsg is privileged and systemd should rate-limit what it writes there after giving priority to more critical messages.

macrael2• 12 years ago

Google+ really doesn't get retweeting, does it. That whole post is written by someone who isn't the one in the photos at the top. A single "originally shared" link points to the original.

Retweeting was a genius move by twitter, a post by someone you don't follow appears in your timeline and it looks like any other and attribution is perfectly captured.

rg3• 12 years ago

Totally agree. As things are currently, wouldn't it be best if the story link was to the actual original post?

https://plus.google.com/+LennartPoetteringTheOneAndOnly/post...

macrael2• 12 years ago

My guess is that the comments on this one led to it being posted here (there is discussion from Linus and others down there.)

sprash• 12 years ago

A kernel is just one component of the OS among many, and ultimately an implementation detail.

-- Lennard Poettering

Jach• 12 years ago

God damn it, now in addition to fighting off pulseaudio and systemd from being on my Gentoo box, in a few years I'm going to have to fight off the Lennart kernel...

antocv2• 12 years ago

Uhm without Linux specific kernel details systemd would not exist.

This as well as journalctl taking minutes to show logs after months of normal usage...systemd is out from my Arch.

notfoss2• 12 years ago

May I suggest OpenRC:

http://blog.notfoss.com/posts/openrc-on-arch-linux/

Gigablah• 12 years ago

"Here's a nice note for all the reporters who are emailing me, go suck it."

"Yes it's titillating, and drives page views, but really, is that the best use of an Liberal Arts degree?"

Ah, the perfect guy to represent our profession.

webreac• 12 years ago

to be clear (I am not english). Ratelimit means drop some logs. Lennart Poettering is commplaining because he wants the kernel to be happy with him flooding the kernel with logs.

When I debug software, I do not want to lose logs and I do not want software to produce so much logs it is impossible to use.

Fasebook• 12 years ago

thankfully we have systemd to manage these kinds of module dependency startup problems in linux now.

ZeroGravitas2• 12 years ago

I wonder how this whole thing would have gone down if the original bug report had simply described the problem rather than demanded their preferred solution to what they thought the problem was.

Linus and others keep acting like the bug was "systemd floods the kernel logs when I pass the debug flag" but it was actually it was "Do not parse "debug" command line ".

The obvious answer to the first is " Sorry about that, the bug is fixed in master branch".

The obvious answer to the second is "No, that's what it's for"

Tobu3• 12 years ago

I think Linus has been pretty clear on both topics (fix A, and if you won't, fix B) and on general policy (don't break it for users or I don't want to work with you in the kernel). Yeah, the reporter could have framed it better from the start, but the developers are supposed to be adults and handle imperfect bug reports too.

FWIW, I don't really consider the first issue fixed. The trigger was lost in a refactoring that can't be backported, and there are probably other ways to trigger the issue.

notfoss2• 12 years ago

I am surprised by the hostility displayed by the systemd guys. I was expecting them to admit they were wrong, and things would be back on track.

With such a reply, I wonder what effect will this have on kdbus making forays into the kernel (not that I am looking forward to it ;).

Response to systemd's use of the debug flag

Related Stories