How to Crash Systemd in One Command

https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_tweet

by agwa • 10 years ago

224 184 10 years ago

agwa.name

tcoppi• 10 years ago

"Systemd is defective by design."

The most succinct explanation of Systemd possible.

mSparks• 10 years ago

I've complained a few times that systemd haters havent explained the hate.

having read this post I'd say a slightly less succinct but infinitely more informative version would be

"Systemd is defective by design because they totally disregard the fail-safe by design rules"

qwertyuiop92410• 10 years ago

Yes. I wish people would explain their hatred. I understand that there are valid reasons to hate systemd, which is why I avoid it like the plague. But so many people get sucked in by the initial usefulness, and don't see the problems lurking beneath the surface.

justin_vanw7• 10 years ago

It seems like it is quite fashionable to hate on systemd, and it seems like systemd is kindof a piece of crap - in some ways.

However, linux is missing basic functionality other os's offer, and systemd is showing up and trying to fill in those blanks. This is open source, if you don't like systemd, if you think it's crap, if you think there are obviously better ways to do it, well, what are you waiting for then?

Systemd is winning because they showed up and basically nobody else did. It's open source fork it and 'do it right'. Or point out the problems and hope that someone else will step up instead?

I think the world would better place if people spent less time griping and more time fighting to get their patch that fixes the problems merged. I guess coding is harder than writing blog posts though.

EDIT: I'm not saying systemd is perfect or even good. My point is that the people writing the code are going to have the final say, whether it's right or wrong, and it is unlikely that people writing blog posts lampooning it are really going to make a difference. I mean systemd has been despised and highly criticized from the beginning, and look where we are.

fapjacks8• 10 years ago

"Systemd is winning because they showed up and basically nobody else did"

I can't see how you would qualify this statement, which is utterly false. Specifically, systemd adoption was the result of political pressure by LP (the author), who works at Red Hat. There are plenty of other choices out there. The idea that systemd won because of some kind of meritocracy is wrong. It was mostly political with some technical discussion at the edges. Most people "making the decision" actually just outsourced their critical thinking to the Debian Technical Committee. That's how we ended up with most distros adopting systemd. Systemd is better than sysvinit (for some values of "better"), and that is what the Debian TC voted on, and what caused them to adopt systemd. Not that systemd is better than other, more modern init systems.

justin_vanw7• 10 years ago

What are the other choices? Upstart was awful, and init is a hundred years old.

georgyo3• 10 years ago

systemd is by far the best linux init system. The options are sysvinit, OpenRC, Upstart, and systemd. Of those, the only reasonable choice is systemd.

Upstart was almost good, but its really frustrating to use. OpenRC is decent, but confusing.

systemd solved problems, not only that it solved them well.

The scope creep of systemd is something that is a bit alarming, however even with the creep it's doing a good job. The tools that it is building work fairly well, and its a relatively new project.

zdw2• 10 years ago

Alternatives have been implemented before:

http://cr.yp.to/daemontools.html http://smarden.org/runit/

The more you look at any of DJB's software, the more of the future you'll see.

qwertyuiop92410• 10 years ago

you forgot http://skarnet.org/software/s6/

ultramancool3• 10 years ago

I've switched everything I write to use runit because runit doesn't require any particular init system, you can easily integrate it with sysvinit, freebsd's RC, ubuntu's upstart, systemd, etc. It just runs on top and doesn't need to be the king and it's incredibly simple to write scripts for it. Completely removes the need to worry about this init mess.

lasermike0262• 10 years ago

The Unix philosophy: Write programs that do one thing and do it well.

Systemd doesn't observe the Unix philosophy and that bothers many people.

geofft7• 10 years ago

It's baffling to see people insist that that's the UNIX philosophy when the UNIX philosophy has always been about monolithic kernels over microkernels, so "do one thing" is out the window, and about worse-is-better over do-the-right-thing (https://www.dreamsongs.com/RiseOfWorseIsBetter.html), so "do it well" is out the window.

And the userspace tools don't follow this either. Quoting http://prog21.dadgum.com/139.html :

The UNIX ls utility seemed like a good idea at the time. It's the poster child for the UNIX way: a small tool that does exactly one thing well. Here that thing is to display a list of filenames. But deciding exactly what filenames to display and in what format led to the addition of over 35 command-line switches. Now the man page for the BSD version of ls bears the shame of this footnote: "To maintain backward compatibility, the relationships between the many options are quite complex."

Why, for instance, does ls have its own sorting options if the sort command exists? If ls, the most quintessential UNIX command, can't do one thing and do it well, why should anything else?

koffiezet• 10 years ago

I'm not that strict on the "unix philosophy". It tends to work, but sure there are exceptions. But if you happen to write something like that does many things, you should also make sure you do every single of those many things well.

What bothers me is that systemd doesn't do that, and that somehow this "unix philosophy" criticism seems to be an excuse/defense for crappy design and code.

qwertyuiop92410• 10 years ago

what functionality? systemd integrated components, it didn't do a whole not new. I'm not firmly convinced that replacing cron was a good idea, integrating udev and the -kits definitely wasn't, and neither was integrating the hostname, or filesystem mounting. And again, I really can't think of anything truly new that it does.

namecast• 10 years ago

I strongly disagree. I didn't need anyone to "show up" to replace my init system in production across several thousand servers, and the way systemd "won" that battle wrt Debian was, to be polite, controversial, political and extremely divisive. They sure as heck didn't win because "they showed up and no one else did". They showed up and RedHat's money showed up with them.

I hold no animus towards the systemd project's goals or aspirations, but I have yet to hear a compelling reason for systemd to be shipping in it's current state.

korethr3• 10 years ago

I hold no animus towards the systemd project's goals or aspirations, but I have yet to hear a compelling reason for systemd to be shipping in it's current state.

I am, in a way, reminded reminded of PulseAudio. I kept PA of my desktop and laptop for years, having heard about PA being a complex nightmare to configure and make work that caused audio to break. Eventually, there was an application I needed to use that had a hard dependency on PA, so I bit the bullet and installed it. And everything magically Just Worked. My only complaint at the moment is that the pactl command seems more fiddly and verbose than it might need to be.

A closer look at the complaints about PA showed that many of them were older. This gives me the impression that PA was in fact a steaming pile that broke audio at one point, but eventually matured into something functional and useful.

And I think this is where systemd is at present: perhaps a good idea, but its implementation is not yet production ready, and should not be the default on production systems. And the project's continued scope creep is working against it in that regard.

rhinoceraptor2• 10 years ago

I think Systemd is winning because it works (in the overwhelming majority of cases), and no one cares about init systems except the vocal minority.

1_2__3• 10 years ago

Can we please just have one OS we use in data centers and servers whose features and development is guided by professionals, not rubes? I'm so sick of people telling me I'm not the target user or some variant thereof. Linux is not OSX or Windows, and not swayed by arguments of the form "most people don't are about that". I do. They should. And the fact that they don't shouldn't result in crappy operating systems.

brainspider2• 10 years ago

However, linux is missing basic functionality other os's offer

Could you please elaborate on what some of those are?

systemd is showing up and trying to fill in those blanks

I haven't really kept on top of all systemd releases, but all I can see is them replacing things that already exist, just doing them in a different way.

I agree the init system replacement was necessary, but I'm struggling to understand why and how it extended to trying to extinguish and replace every core function of the operating system ecosystem.

That's rubbing a lot of people the wrong way, especially as mentioned in the article, those who adhere to the unix philosophy of do one thing and do it right.

justin_vanw7• 10 years ago

I basically agree with everything you're saying, I just think if the people who think systemd is misguided spend their time writing blog posts about it, nothing will change and the people actually writing the code will have the final say, right or wrong.

korethr3• 10 years ago

However, linux is missing basic functionality other os's offer, and systemd is showing up and trying to fill in those blanks.

I'm curious what functionality other OSes had that Linux did not before systemd came along. It seems to me that it has been subsuming functions of other systems on Linux that were already present, as opposed to offering anything new.

Socket activation? inetd/xinetd. Timed start of programs? Cron and At Logging? rsyslog/syslog-ng/etc. DNS? -- dnsmasq, powerdns, bind, etc.

Honestly, I think if system had stopped at doing init, and focused solely on that and doing the best damned possible job of it, there'd be less hate. I think what's pissing off a lot of the systemd haters is the scope creep of the project has undergone. I was personally fine with the idea of systemd when all it was promising was init and service management. But then oh, were going to take over /dev, and cron and logging, and logins and DNS and (to be determined). Ahh, thanks, but no thanks.

bjourne4• 10 years ago

I'm curious what functionality other OSes had that Linux did not before systemd came along.

Uniform daemon/service handling. Just look at the scripts in /etc/init.d (or wherever they are located in your particular SysV init setup). It's a total mess.

The rest of what systemd offers I can't comment on because I'm not familiar with it. But it is clear to me that systemd's declarative .service files are a billion times better than shell scripts.

djsumdog3• 10 years ago

We don't live in the open source world of the 1990s. People don't make this stuff in their basements. What was a community of devs that made things in their own time has turned into a huge industry supported system of semi-bullshit sharing. The OSS devs of the 90s wanted open source end products: a GIMP that could replace Photoshop. A blender that could replace 3D Studio. Today we mostly have OSS middleware.

As the other comments have stated, there have been several systemd drop in attempts, such as this one:

http://uselessd.darknedgy.net/

But maintaining them is terribly difficult when Redhat is pouring money into people to work on it full-time. There are lots of systemd-free forks, including ones for Debian and Arch. But you need a large user base to continue to use these forks so other people will be encouraged to support them. Money, donations and good ole kudos help too.

TL;DR we don't live in the open source world you think we do.

jude-3• 10 years ago

Uselessd has been abandoned, as your link points out. The rationale [1] is, predictably, that "mangling systemd into something a bit saner turned out to be frustrating" and it was "a fool's errand" to have tried.

[1] https://forums.darknedgy.net/viewtopic.php?id=4963

wmf• 10 years ago

Is Red Hat funding over 50% of Linux development? If systemd was so bad it seems like maybe our corporate overlords like IBM, Google, Intel, or Canonical could have spoken up or even banded together to compete with it, but they didn't.

gaius• 10 years ago

Systemd is winning because they showed up and basically nobody else did.

Systemd is "winning" because Red Hat are throwing money at it as a way to control the ecosystem.

sparewalking2• 10 years ago

Red Hat refused the project initially.

thescribe2• 10 years ago

I'm not sure forking would be worthwhile, I think the issue people have with systemd is one of goals and structure of the operating system. Any fork that satisfied the other side would not even be in the same category of software.

milankragujevic• 10 years ago

Archived page, since it's not loading for me... http://archive.is/8xUSY

mavdi• 10 years ago

How to crash a blog with one submission

agwa2• 10 years ago

My hosting provider thought I was under DDoS attack and started aggressively throttling port 443. I failed over to my backup provider, probably permanently.

belovedeagle2• 10 years ago

This is an excellent writeup explaining without hysterics why systemd is terrible software. Unfortunately, I expect that this post will be brigaded by systemd-apologists anyways, denying newcomers to the debate the opportunity to hear the facts...

andrewstuart22• 10 years ago

I'm not sure how this comment is helpful to the discussion. As far as I can tell, you'd rather only hear the facts that fit your opinion of systemd as terrible software. I think the obvious right answer is that both sides present only facts and let newcomers decide based on that.

fapjacks8• 10 years ago

I completely agree with you. That would be nice, if people were presented with facts and a choice. But they weren't, and aren't.

belovedeagle2• 10 years ago

As the other reply said, the problem is that newcomers aren't allowed to hear the anti-systemd facts, as the systemd gods have already decided that all Linux users (or at least all distros) must bow down and accept them, and no dissenting opinion is allowed to stay on the front page of HN for long (this post has already been brigaded off the front page, apparently).

geofft7• 10 years ago

I've been thinking for a bit that it's reasonably easy to write a PID 1 that reaps children and sends siginfo_t structs over a pipe to PID 2, which can be an init system as monolithic as you want.

Of course, it would also be simple for the kernel to allow PID 1 to die/crash, and just restart it instead of panicking. I don't know why the kernel continues to treat PID 1 so specially—if the ewontfix.com PID 1 is sufficient, it's simpler for the kernel not to bother with the reparenting logic and stop treating PID 1 as a special process.

mschuster91• 10 years ago

I don't know why the kernel continues to treat PID 1 so specially

By design all processes inherit from PID 1 (directly or indirectly), so you always have an unbroken tree from every process back to PID 1. PID1 aka init is supposed to do bookkeeping (e.g. zombie processing), and if PID1 crashes the bookkeeping records are lost.

Because of this, it is better to fail hard, instead of allowing for corrupted process trees or similar.

Also, ps, top and basically every program that works with processes expects one root process for the tree. Breaking this would basically break compatibility with every existing userland (which the kernel cares the most about!)

justinsaccount7• 10 years ago

Breaking this would basically break compatibility with every existing userland

By "Breaking this" you mean "fixing the fact that pid 1 can't crash and restart"?

Because of this, it is better to fail hard

It can't be fixed because it would "break compatibility" where that "compatibility" is that the machine should panic?

You're not really making a good case for why this can't be changed.

Why not

  * pid 1 exits
  * kernel re-executes init as a new pid 1
  * userspace continues on as if nothing happened.

5ilv3r4• 10 years ago

Ran as a normal user on debian jessie, and now ssh logins are delayed by 30 seconds. I don't see any other problems yet, but this is a VM host and so leave it running for a while for fun. :)

aorth2• 10 years ago

Ran it on an up to date Ubuntu 16.04 and lost the ability to run any systemctl commands like status, restart, etc. Web server and other system services were still working, but I couldn't `reboot` or `systemctl halt` etc... had to hard reboot the VPS from hosting console. :)

andreyv• 10 years ago

"systemctl reboot --force --force" should work in such cases. The reboot will be executed by systemctl itself.

rcarmo• 10 years ago

First they came for PID 1, and we were silent.

Oh, wait, we weren't. But although I don't miss the rickety init scripts, I've grown more and more worried about stuff like this, enough to ponder replacements (in Go or Python, which seem to be generally acceptable for systems software these days).

Ignoring for the moment the usual biases against Python, how feasible/how much support would it be/have to build something to replace systemd (by having PID 1 do the minimum possible, in C, and having the rest done with Python)?

djsumdog3• 10 years ago

I've been working at a Docker shop for a while and I have a feeling it will be the future for running services. It makes systemd somewhat irrelevant.

There are still a lot of issues I've had with Docker, which I won't go into. It also can't replace everything running on your Linux system. But it's a neat idea and big players use it in production for huge workloads.

Now the whole AWS-containers, docker-machine, swarm, mesos, coreos, kubernetes thing is a cluster fuck of choices .. but you have choices. And unlike running an Ansible script on some provides base image (which is totally different from Rackspace, Linode, etc.), you are guaranteed your container will work on any docker engine with little modification (your orchestration files may change though).

I have a feeling, for better or worse, containers will be the future and will stick around long after systemd hopefully dies.

digi_owl8• 10 years ago

Nah, systemd is angling to supplant docker for containers.

There is already a great deal of bad blood between the two because of the reworked cgroups management.

And there is also the likes of CoreOS that is basically systemd-the-distro.

Never mind that we have the likes of xdg-app/flatpak that is basically aiming for what you are thinking about, and it is systemd dependent.

packetized• 10 years ago

Erm, this does not actually seem to work on Centos7, running systemd-219-19.el7_2.13. Even tried it as root inside of a sandbox - nothing.

lake99• 10 years ago

It's the same on Archlinux, systemd-231. Nothing happened. I was able to start and stop a couple of services as quickly as before.

SysArchitect3• 10 years ago

  while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done

Try this. As soon as you loop it, the bug is hit pretty quickly.

nilved• 10 years ago

I don't understand why that makes a difference if the bug is `assert(n > 0)`

agwa2• 10 years ago

Hmmm... I've repro'd on v215, v229, and v230, although only on Debian and Ubuntu. The bug has been in systemd upstream since v209 as far as I can tell by reading the code.

xemdetia• 10 years ago

At this point in time my biggest frustration is that even compared to upstart writing concise and clear config files for network services is just really hard, and there isn't great documentation or a short book around that covers all the edge cases other than TIAS. I feel like I've wasted more time trying to get a tidy systemd configuration than I ever had even with upstart or some of the other ones.

bitmapbrother• 10 years ago

Despite the banality, the bug is serious, as it allows any local user to trivially perform a denial-of-service attack against a critical system component.

Is it really that serious of a bug? You have to be a local user in order to execute it. Are other OS's somehow resilient to the actions of local user to degrade the OS?

The Linux ecosystem has fallen behind other operating systems in writing secure and robust software. While Microsoft was hardening Windows and Apple was developing iOS, open source software became complacent.

He must be unaware of the numerous patches Microsoft and Apple issue on a regular basis to fix their "secure and robust software".

falcolas• 10 years ago

Any remote code execution bug can exploit this. Normally, a remote execution code bug when the user is non-root isn't a huge issue, because the user effectively sandboxes them away. This bypasses all of these benefits.

Containers may prove to be a sufficient sandbox, assuming this is not executable against kdbus (which is shared across containers since it's in the kernel).

redbeard0x0a• 10 years ago

This article got me wondering if maybe the NSA or similar has somebody on the inside of the systemd project that is helping the project along.

Systemd has replaced a lot of init systems in the linux ecosystem and the general development practices of the project is leaving a huge footprint of code that might be exploited.

If they aren't involved, I'm sure they probably know a few different exploits for the system already.

pdkl953• 10 years ago

If you want to go down that rabbit hole, see:

https://igurublog.wordpress.com/2014/02/17/biography-of-a-cy...

johnfjacobi• 10 years ago

I've heard this a lot and first just dismissed it as a conspiracy theory. Is there actually any substance to these thoughts, or are they only speculation?

xaduha2• 10 years ago

Don't really care about that bug, it will be fixed I'm sure. But thank you for this post anyway, the word LLMNR caught my eye, very nice to have if it works.

EDIT: works fine with Windows, without installing Bonjour/iTunes.

SysArchitect3• 10 years ago

The issue has a bug report on the systemd repo on Github: https://github.com/systemd/systemd/issues/4234

walter_bishop• 10 years ago

Doesn't work here on Ubuntu 16.04 xenial

Twirrim2• 10 years ago

I've put a comment on the github bug with an updated PoC. IT does work on 16.04, but you seem to have to do it over and over until it triggers. A working PoC seems to be:

while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done

zzxxmm11aajj• 10 years ago

Note: 112 comments from the peanut gallery before someone looked at the code enough to submit the trivial fix.

m45t3r2• 10 years ago

You could simple do a DoS attack with a local user using a fork bomb too. It would be even shorter than the command showed in this post blog.

So while this is a bug, the author is overreacting just to make systemd look bad.

rascul• 10 years ago

your fork bomb can be mitigated with ulimit

m45t3r2• 10 years ago

I am not saying that this is not a bug, just that this isn't the problem that the author is making. If you have local access, you probably can do other things if your only objective is to do a DoS.

aorth2• 10 years ago

A CVE has been requested, albeit after publication of the bug:

http://www.openwall.com/lists/oss-security/2016/09/28/9

JdeBP7• 10 years ago

Discussed on Hacker News at https://news.ycombinator.com/item?id=12600994 .

wyldfire• 10 years ago

The immediate question raised by this bug is what kind of quality assurance process would allow such a simple bug to exist for over two years (it was introduced in systemd 209). Isn't the empty string an obvious test case?

All bugs are terribly easy to spot in hindsight. I don't think it's fair to criticize this one in that fashion. Systemd has lots of pros and cons, but the age of this bug isn't interesting IMO. Its severity is, but that only speaks to the critical need to patch it ASAP.

JdeBP7• 10 years ago

It's interesting to see a "hacker" discussion of this that entirely misses several topics that one would expect programmers to at least mention:

* Whether release builds should have assert() enabled.

* Whether a process #1 program should use assertions at all.

* Whether assert() is the right mechanism to use for input validation.

* How a process #1 program should recover from assertion failures.

fridsun7• 10 years ago

Lennart Poettering has replied on Reddit.

https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_...

So let's summarize this. There's a bug in some software. OMG! Shock! This of course never happened before!
The bug is in not exploitable remotely. The bug does not allow privilege escalation nor insertion of code. This of course makes the bug a massive vulnerability like there was no other on the planet ... ever. As bad as heartbleed multiplied by the Debian OpenSSL random generator bug to the power of 10.
The bug is caused by an error check that filters out garbage sent to PID 1. The check works correctly, except that the resulting action is a too harsh: instead of complaining and dropping it will abort the process. Such a bug is of course unprecedented and the authors of said software should be stoned and flogged right away given the severity of the issue: after all a safety check worked a bit too well, and we really can't have that because undefined behaviour of course would be a lot better than a local DoS.
The project the error was found in is large. Yet the number of CVEs collect so far is pretty small comparing it witht other projects of similar extent. Given that another bug was discovered now this obviously shows how incompetent the programmers are and that security is a unknown concept to them.
The program the bug was found in is longer than 50 lines of code but runs with privileges, all written in a low-level programming langauge that many call little more than a fancy macro assembler. The code runs on top of an operating system kernel written in the same language but running with a lot higher privileges and consisting of expoentially more lines of code including drivers of questionnable quality. This together is of course proof that the project at hand is flawed conceptually to its core.
Dha!
Lennart
(More seriously: yes this is a bug, we should fix it. But it's very low impact and the bruhaha it generated appears wildly out of scale. If all bugs in the wider Open Source ecosystem would have a similarly low impact we'd live in a much much safer world!)

JdeBP7• 10 years ago

Lennart Poettering's response to this was interesting:

The bug is caused by an error check that filters out garbage sent to PID 1. The check works correctly, except that the resulting action is a too harsh: instead of complaining and dropping it will abort the process.

* https://www.reddit.com/r/linux/comments/54yfcd/how_to_crash_...

In fact, the opposite was the case. The root of the problem was identified in the GitHub bug report:

* https://github.com/systemd/systemd/issues/4234#issuecomment-...

* hhttps://github.com/systemd/systemd/commit/d875aa8ce10b458dc2...

What happened was that Lennart Poettering removed an error check that filtered out zero-length messages. This left the flow of control to fall through to a later point where an assertion, that had been earlier added by Lennart Poettering with the assumption that this check for zero was in place (as it had been at the time), then triggered.

Of course, no-one that I have seen has yet asked why "" as a command-line argument to systemd-notify results in a zero-length message in the first place. After all, according to the doco that would be length 1, a single terminating LF byte, not zero length. That said, the server still should be proof against zero-length network input.

How to Crash Systemd in One Command

Related Stories