The bad side of systemd: two recent systemd failures

http://utcc.utoronto.ca/~cks/space/blog/linux/SystemdCrashAndMore

by lreeves • 12 years ago

121 109 12 years ago

utcc.utoronto.ca

I'm no fan of systemd, but do we really need to be nitpicking its every failing? The alternatives are certainly not perfect either. If you don't like it, don't use it. Many distros are still committed to supporting other init systems along side systemd and many BSDs have a very good philosophy with things like this and may be worth a look too.

Can we just agree to move on? I'm sick of the endless banter. If you like systemd, use it, if you don't like it, don't. If you think it needs more time to mature, stay away from it for a while and experiment occasionally. That's my plan at least.

broodbucket• 12 years ago

Within all the noise about systemd I think posts like this are quite beneficial. Instead of discussing politics or throwing out buzzwords, it presents a specific technical shortcoming. Positive or negative, these are the types of things that people should be reading about in relation to systemd.

The problem is that people are deciding whether or not they like systemd based on "do you like Lennart" or "do you like The UNIX Philosophy", because that's what people are making noise about.

Mister_Snuggles8• 12 years ago

As I said in another comment, I really wish it went into more detail on the actual problem.

The problems with journalctl are usability ones, which doesn't make them any less of a problem, but doesn't point to an underlying technical failure either.

The problems with systemd are basically stated as "a Fedora upgrade causes systemd to go bad." Is this a systemd problem or a Fedora problem?

felixgallo• 12 years ago

those are completely valid criteria. What's wrong with wanting your system to follow the closest thing to a known axiom in operational computer science that has ever existed?

noselasd• 12 years ago

I don't really have a good or bad opinion about systemd, to me the init system and things around it are just pieces that needs to work.

The points discussed here are serious, should be filed as bug reports, and should be acted on by the developers. But the article also just sounds like systemd bashing, software has bugs and poor thought out solutions - which can be fixed or improved - it's just code.

I could probably have written a similar article about many, many Fedora (and Debian) upgrades over the course of a decade or so about core components such as distro upgrades making the init system crashing, failing to start services, glibc upgrades screwing up servers, grub upgrades calling for a 2 hour car drive to fix and so on. But I didn't, I either filed a bug report, fixed the code, or was just too plain busy getting things back to working.

vezzy-fnord2• 12 years ago

Given the huge ambitions of the project maintainers and how everyone's rushing to adopt it even amidst sloppy integration work, yes, the criticism is worth it.

"The alternatives are certainly not perfect" is no consolation. At all. Especially considering the vast majority were never fairly considered to begin with.

ultramancool6• 12 years ago

I had the same sort of annoyances, but instead of whining, I simply ditched it for now. I don't see why people have such a problem doing this.

e40• 12 years ago

if you like systemd, use it, if you don't like it, don't.

That is so naive. No one chooses to use systemd. People choose a distro. It's being forced on long time users of distros without any of the users having a say.

Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound.

ansible2• 12 years ago

Yeah, I can switch from one distro to another. That's a choice I have. But it's not so simple as you make it sound.

Switching distros is like moving to a new city. If you stay in the same state (or at least in the same country) it isn't too disorienting. But switching to something completely different (like on another continent, or going from an RPM-based to deb-based distro) will require some significant effort to adapt to the new situation.

We pick our distro of choice based on a number of factors. Just because it has some downsides doesn't mean we have to move immediately. But that doesn't mean we can't give (constructive!) feedback if there are problems.

vidarh• 12 years ago

No one chooses to use systemd.

I did. I've been systematically installing systemd on my Debian machines.

chimeracoder• 12 years ago

It's being forced on long time users of distros without any of the users having a say.

It's being adopted by distributions in accordance with the stated and implied wishes of the developers, maintainers, and users of the distributions.

The reason that (e.g.) Arch dropped support for legacy init so quickly was because they literally failed to find anyone who wanted to go through all the work of maintaining the old init system. If none of the users want to do the work, then nothing's being "forced" - that's the way community-organized FOSS projects work![0]

Same goes for Debian, who made the switch in accordiance with their normal development process (including a general resolution in support of systemd!). Debian has promised to maintain systemd-shim (supporting other init systems) through at least Jessie, and possibly longer, depending on whether there is enough demand for alternative init systems, and support from developers for maintaining it in future releases.

Ironically, the biggest threat to systemd-shim in Jessie's successor are people who claim they want to fork Debian - if all the people who hate systemd leave Debian, then nobody who wants to actually do the work to maintain the alternative init systems will remain with the Debian project[1].

[0] Even then, you can still find an AUR package for it (https://aur.archlinux.org/packages/initscripts-fork/), though as with everything in the AUR, it's subject to much less testing. Note that the package has only 27 votes - this means that only 27 people actually said that they care about moving this package back into the main distro.

[1] This may be a bad or a good thing, depending on who you ask.

zanny• 12 years ago

Because its a free software project you are not paying for and use as a gratis gift from the developers and maintainers of the distro.

And those developers and maintainers wanted to use systemd.

If, as a user, you disagree with the developers and maintainers of your choice distribution, you picked the wrong distro, and should either find developers and maintainers that align with your views or fork it and become a developer / maintainer yourself if nobody provides for your use case.

You cannot bitch on a high horse when not paying a cent for any of this about how the developers and maintainers are doing what they want with their distro. Well, you can, but nothing will ever happen - they already made their choice based on their needs and use cases.

zobzu• 12 years ago

Not sure if /sbin/init failing is a nitpick. its kind of a catastrophic failure as far operating systems are concerned.

digi_owl8• 12 years ago

Never mind that it results in a zombie state where you still have a shell, but no commands actually work.

howeyc2• 12 years ago

Can we just agree to move on?

No, we can't. Sorry but that's just how the world works. Just like every Go language post has "generics or crap" mentioned, every systemd post will have "The Unix Way" mentioned.

A price of freedom of speech is having to deal with self-filtering idiotic viewpoints. Worth it in my opinion, but I'm an anti-censorship for EVERYTHING person.

ultramancool6• 12 years ago

I'm not suggesting censorship of any sort. I'm certainly the same way. I'm just suggesting level-headedness and a willingness to take or leave it and move on with life.

rdtsc• 12 years ago

If you don't like it, don't use it.

Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either.

I'm sick of the endless banter.

Ok but "use something else" or "fork it" is not exactly solving the problem either.

digi_owl8• 12 years ago

And thats the thing. If it was "just a init", it would be just as replaceable as sysv, openrc or runit. But it is not, so it is not. As best i can tell, a major argument for going systemd is that something more user facing (Gnome) depends on a subsystem of systemd, and that in turn requires that systemd is used as init. In other words, systemd as a project is invasive and insidious.

EmanueleAina5• 12 years ago

Have you tried removing systemd from latest Ubuntu? I didn't go very well for me. I imagine removing it from RHEL 7 won't go well over either.

Ubuntu is not the Universal Operating System, that's Debian. This means that Ubuntu is definitely opinionated in its software selection, and rightly so. You couldn't replace Upstart and now Ubuntu relies on systemd. If you have the skills to fully evaluate the costs and benefits of mixing and matching random pieces of software, distribution like Debian or Gentoo may be a better fit for you.

Indeed, switching to sysvinit in Debian is trivial and the systemd team worked really really hard to make sure the transition is smooth even if one keeps switching back and forth. They should really be praised for the effort they've put in it.

_cipher_• 12 years ago

I'm no fan of systemd, but do we really need to be nitpicking its every failing?

Yes. When some developers are pushing down your (and mine) throat critical software that sucks _that_ much, that's the least someone can do.

EmanueleAina5• 12 years ago

Some developers wrote some code and published it in a software package under a free software license.

Other developers picked up said package and added it to the package collection they are maintaining.

You choose to use the package collection maintained by said developers, which means that you trust their technical decisions otherwise you'd be running software from people you don't trust, which is not a great idea.

Nobody is pushing software down anyone's throat. Please avoid such hyperbolic mischaracterization, it really does not contribute anything to the discussion.

skywhopper• 12 years ago

I think ongoing constructive discussion about the technical and philosophical issues with software design, particularly of open-source software that affects huge numbers of users, is always valuable. The OP here even likes systemd, so the article isn't nitpicking.

As others have pointed out, when your init process faults, that's a major problem. In the real world, for the core elements of an operating system, whatever features they have are less important than how well they fail. In this case, the OP has identified a very very bad failure case.

More importantly, this report makes concrete the theoretical complaint against systemd that its monolithic design is not just a philosophical issue, but a practical one. The init process needs to be rock solid, but a law of software is that bigger software is less stable. So if the sprawling nature of systemd means it will tend to become less stable over time, then we have a major practical issue that needs to be addressed.

This is not a matter of not moving on. Rather, given that systemd has been adopted by several major Linux distributions, it's actually far more important now to criticize its design failures, to draw attention to get things fixed. If "moving on" means we shouldn't critique the software, then how will it ever improve?

mrweasel• 12 years ago

If you like systemd, use it, if you don't like it, don't.

I think people would care less about systemd if that was a realistic choice. I strongly believe that the whole systemd debate would go away if systemd wasn't a dependency of GNOME and other stuff.

Personally I have no strong feelings one way or the other, but I do fear that the BSDs are going to be left behind as they to some extend have been piggybacking on Linux in terms of desktop software.

exabrial• 12 years ago

you're missing the point... SystemD is about RedHat trying to take over the entire Linux ecosystem. It's intentionally designed so if you use any part of it (which there are good parts) then you have to use all of it (which there are many bad parts). Oh and conveniently RedHat is the main project maintainers.

XorNot3• 12 years ago

I can't speak to the first problem, but the second is more of "this is strange and I don't care to learn how it works".

journalctl --no-pager

Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options.

masklinn2• 12 years ago

I can't speak to the first problem, but the second is more of "this is strange and I don't care to learn how it works".

I can't speak for you not speaking to the first problem, but the second is more of "this is strange and I didn't care to read the text"

In the grand systemd tradition, there is no option to control this; all you can do is force journalctl to not use a pager or work out how to change things inside the pager to not do this.

(emphasis mine)

Problem solved. I mean, you have the command, it's right there in the usage text. Which, awesomely, does invoke a pager when you do journalctl --help so if you're stuck on a 80x25 console (as you might be on a broken system) you can easily actually read the help options.

From reading the text, I'm under the impression that the issue is the mangling of $LESS. One would expect that $LESS is set up how the user wants it, if the user wants -S he'll set that.

darklajid8• 12 years ago

Okay, okay. I understand that the user might be surprised.

man journalctl

    $SYSTEMD_LESS
           Override the default options passed to less ("FRSXMK").

So - whereever the user defines $LESS I'd add a line a la

    export SYSTEMD_LESS=$LESS

and that should solve the problem nicely, right? Surprising? Not intuitive? Maybe. But easily fixable with a quick peek at the man page.

regularfry• 12 years ago

Wait, so the --no-pager does invoke the pager, but with different options? That would be surprising.

It looks to me like you either get a pager with broken settings, or you get no pager at all.

mdpye2• 12 years ago

While I agree with your first paragraph,

if you're stuck on a 80x25 console

What's wrong with `journalctl | less` in this case? The only command I use regularly which invokes a pager by default is `man`, and that seems like an actual special case.

Edit: FTR, I agree with your assessment in the first paragraph, but not the judgement I infer you passing. IMO, things should avoid being strange.

masklinn2• 12 years ago

It can make sense to invoke a pager by default to avoid blowing the current terminal scrollback to smithereen if the user himself forgets it. Especially for diagnostic tools, where the information left in the scrollback buffer could be useful and not saved anywhere.

cesarb4• 12 years ago

The only command I use regularly which invokes a pager by default is `man`

Git also invokes a pager by default. And it also sets $LESS when doing so. In fact, it wouldn't surprise me if systemd's automatic use of a pager was based on git; IIRC, it used to set $LESS to the exact same value git used (I believe systemd has changed it since then).

XorNot3• 12 years ago

I tested this just now: journalctl | less works normally for me on Ubuntu Utopic.

darklajid8• 12 years ago

Or..

       -a, --all
           Show all fields in full, even if they include unprintable characters
           or are very long.

       -f, --follow
           Show only the most recent journal entries, and continuously print
           new entries as they are appended to the journal.

(I'm talking about -a, copied the _following_ -f because that is what the author is trying to use here. It was right there..?)

Edit: Although I'm not sure if he's complaining about truncated fields or long lines. Hmm..

jeremy7600• 12 years ago

"In the grand systemd tradition, there is no option to control this; all you can do is force journalctl to not use a pager or work out how to change things inside the pager to not do this."

He mentioned that. So, yes, it looks like he cared to learn how it works.

XorNot3• 12 years ago

But apparently not enough to think to alias journalctl --no-pager, like is the advice for literally dozens of other Unix shell commands? Because colorized ls output is certainly not an "option" either in that sense.

Mister_Snuggles8• 12 years ago

I found that when the output is piped it doesn't invoke the pager. This seems reasonable to me.

What doesn't seem reasonable is the truncation behaviour. When the output isn't piped, it gets truncated to the width of my terminal. When the output is piped, it gets truncated to 80 columns. I'd expect no truncation when the output is not a terminal.

This is an outdated systemd though (I installed it on a Rasberry Pi [Raspbian] in order to play with it), so this may have changed.

patzerhacker• 12 years ago

I found that when the output is piped it doesn't invoke the pager. This seems reasonable to me.

This might be less being smart and getting out of the way. From http://en.wikipedia.org/wiki/Less_%28Unix%29#Usage:

By default, less displays the contents of the file to the standard output (one screen at a time). If the file name argument is omitted, it displays the contents from standard input (usually the output of another command through a pipe). If the output is redirected to anything other than a terminal, for example a pipe to another command, less behaves like cat.

lstyls• 12 years ago

How much of the blame should systemd be taking for these bugs? Unless the same segfault is happening on other distros, this seems like a problem in Fedora rather than systemd.

I suppose this does give some substance to the systemd criticisms that it is too system-critical for a process that is technically outside of the Linux kernel.

kosinus• 12 years ago

If you're saying, a bug should always be investigated downstream before reporting it upstream, then I agree.

If you're saying, there are scenarios where segfaults caused by upstream code are not upstream bugs, then I disagree.

Mister_Snuggles8• 12 years ago

It's interesting to see examples of where systemd falls down, though I'd be interested to hear what the actual problem is. The Fedora bug report and linked page basically say "systemd fell over during an upgrade", but is this really a systemd problem or is it a Fedora problem?

digi_owl8• 12 years ago

At this point in time they seem joined at the hip. I think a Pottering retort about Debian users complaining about the size of systemd was that it was introduced over time on Fedora. So if any distro should make systemd shine, it would be Fedora.

cesarb4• 12 years ago

It's failing during an upgrade. My guess would be that it's a systemd problem, but one triggered by mixing systemd versions. Probably the systemctl from Fedora 21 does something that the systemd from Fedora 20 doesn't like.

If that's the case, the bug might have even already been fixed in systemd upstream, but the fix wasn't applied to the older systemd in Fedora 20. The solution would be to backport the fix to Fedora 20's systemd package, so users who do a "yum update" before upgrading to Fedora 21 would have it.

An alternative workaround might be to, immediately after unpacking the new systemd, send a SIGTERM to PID 1. According to http://www.freedesktop.org/software/systemd/man/systemd.html, that signal directly tells PID 1 to re-exec itself, without using systemctl in the process. That would make PID 1 be the new systemd before anything tries to use the new systemctl.

siebenmann• 12 years ago

Almost any time PID 1 segfaults, it's PID 1's fault. With PID 1 being systemd, that makes it systemd's fault here. Based on gdb stack backtraces in the Fedora bug report and looking at the systemd code, it seems like some sort of memory corruption or overwrite, perhaps a use after free issue. The segfault itself comes from dereferencing a clearly invalid pointer and said pointer was obtained by dereferencing a structure field through another pointer, so you'd get exactly this result if the structure was overwritten with other data at some point.

(In my grumpy sysadmin view, it is PID 1's fault even if the distribution is doing odd things around PID 1. Init processes need to be absolutely rock solid and extremely defensively coded, precisely because the world basically dies if they ever fall over.)

(I am the author of the original post.)

digi_owl8• 12 years ago

What i find interesting is that it is the traditional sysadmins that find fault after fault with systemd.

On the other hand those that embrace it come from web/cloud development, with an eye towards virtualization and containerization.

I think someone somewhere once compared it to pets vs cattle.

Traditional servers are the pets of sysadmins. Groomed and cared for to make sure they never keel over unexpectedly.

To the cloud "admins" (or maybe i should call them devops?) servers are cattle. they have X of them living in some cloud "farm" somewhere, and can order any number of them to be slaughtered and replace at the drop of a pin.

darklajid8• 12 years ago

The first problem, PID 1 crashing, is certainly worth complaining about.

The rest seems mostly nitpicking. I wouldn't work myself up about the pager issue as discussed elsewhere here. I do somewhat agree with a rather useless default for the range of things to show (i.e. I'd prefer -b [1] to be implied unless specified otherwise, for example).

1: -b means 'since the last boot' for people not using journalctl. The default is instead to show the full journal from its beginning, as stated in the article.

raverbashing• 12 years ago

" Oh, the full message is available, all right, but journalctl specifically and deliberately invokes the pager in a mode where you have to scroll sideways to see long lines. Under no circumstance is all of a long line visible on screen at once so that you may, for example, copy it into a bug report."

Well, that's what you get by supporting a system built by obnoxious incompetents that think they can replace the most important process in the machine

"(Oh, and journalctl goes out of its way to set up this behavior. Not by passing command line arguments to less, because that would be too obvious (you might spot it in a ps listing, for example); instead it mangles $LESS to effectively add the '-S' option, among other things.)"

Yes, of course.

wtbob• 12 years ago

Regarding the first problem, as someone else said, 'Remember how awesome PulseAudio was? Imagine that for PID 1!'

cbd1984• 12 years ago

PulseAudio gives me a better out-of-the-box experience than any other sound system, bar none.

eots• 12 years ago

Guess I'm the only one who had a cron job restarting the PulseAudio process regularly.

Wilya2• 12 years ago

That might be true today (I no longer have linux desktops, so I have no idea), but it took years of buggy (or extremely hard to configure) releases to reach that point.

upofadown• 12 years ago

I felt dirty, but this made me laugh out loud:

While I'm here, let me mention that journalctl's default behavior of 'show all messages since the beginning of time in forward chronological order' is about the most useless default I can imagine.

cesarb4• 12 years ago

Yeah, that's an annoying default. At least it also uses a pager by default, so it stops after the first screenful.

On the other hand, "journalctl -b" (show all messages since the machine has been last powered on, in forward chronological order) is quite useful.

the_why_of_y2• 12 years ago

Yep... even better is that it takes an index, so you can "journalctl --list-boots" to look up boot by timestamp and "journalctl -b -2" to look at the log of the second-to-last boot to copy and report the kernel oops in there.

userbinator• 12 years ago

Looks like there is another bug open already with a similar backtrace, also triggered by an upgrade:

https://bugzilla.redhat.com/show_bug.cgi?id=1130633

staticshock• 12 years ago

Guys, the `less -S` problem is bunk, because you can easily change it at runtime. From the OPTIONS section of `man less`:

Most options may be changed while less is running, via the "-" command.

So, just start `journalctl -l` as you would, then type in `-S` and scroll the screen in any direction to force it to re-draw.

Discussing it any further is bikeshedding, and distracts from the much more interesting segfault discussion.

deathcakes• 12 years ago

To which I would say that you are completely missing the point of the complaint levelled here - that you can work around a stupid design decision does not negate the stupidity of that design decision. Personally I think the default of show all logs ever is worse but both mean that what should be an integral and useful tool is initially off putting at best and constantly irritating at worst. I say this as an arch linux user for the past three years.

roller• 12 years ago

With apologies for drifting off topic:

I love less's command line interface for some of this stuff.

The command line switch to chop long lines: -S The in-app command to toggle chop lines: -S

The reverse is almost true for things like /search, F (follow) or G (go to end). They can be used from the command line with +/search, +F or +G

angersock• 12 years ago

Man, anybody else remember being able to use tail and grep to decode system logs?

And not having the init process crash?

Seriously, this is clownshoes. Their developers and evangelists should get hit by a SIGBUS.

jdlyga• 12 years ago

All I've been seeing lately are posts very critical of systemd. I really don't know much about it, and I would love to hear the other side of the story. Why is it so popular lately?

The bad side of systemd: two recent systemd failures

Related Stories