Systemd as tragedy

https://lwn.net/SubscriberLink/777595/a71362cc65b1c271/

by wyldfire • 7 years ago

288 401 7 years ago

ordu2• 7 years ago

"It's software" so of course it's buggy, he said. The notion that systemd has to be perfect, unlike any other system, raises the bar too high.

systemd is a PID 1 program, it means it have to raise bar higher. When troubles begin, you would need tools to fix them, and if PID1 is crashed, you are out of luck. If system cannot boot into shell, you'd need to fix it from initrd shell. Or to boot other system, to fix this one. It sucks.

Linux kernel chases very high standards of reliability, because when kernel panics it is even worse than PID1 crash. Init system should follow the same standards as linux.

geofft6• 7 years ago

Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me. Lots of other stuff, including the kernel itself, has crashed.

The bar is higher for pid 1 - if I were designing systemd I would have made a tiny pid 1 that just did message-passing to a more complex secondary process that could be restarted, or something, just to be safe - but I think systemd has empirically cleared the bar.

CBLT• 7 years ago

I've never experienced an outright crash, but I've been bitten by [0] on some of my servers.

[0] https://github.com/systemd/systemd/issues/719

rachelbythebay• 7 years ago

Yep. CentOS 6's upstart can be felled by generating a bunch of inotify events in /etc.

http://rachelbythebay.com/w/2014/11/24/touch/

rurban2• 7 years ago

The current debian testing version crashes with a NULL pointer segv in the kernel module. You need to downgrade to the previous version.

ordu2• 7 years ago

Have you ever had pid 1 (systemd or any other init) crash?

No, I have not. But I have seen how systemd gracefully failed to boot system to login, with good looking colorful error message. Something that reminded me "Keyboard is not found. Press F1 to run setup."

ploxiln• 7 years ago

I've had shutdown (for reboot) hang a few times after a systemd update, forcing me to cut the power. It's made me a bit paranoid, so I block the systemd package from having updates automatically installed, and every 6 months or so carefully manage update and reboot of each and every server ...

EDITs:

there's the classic case of the linux "debug" parameter: https://bugs.freedesktop.org/show_bug.cgi?id=76935

and the even more classic case of firmware loading events: https://lkml.org/lkml/2012/10/3/484

and while "all software has bugs" systemd really has the most annoying bugs (by virtue of trying to do everything core to the system) and always insists that they are features and we are backwards whiny geeks for complaining.

65a• 7 years ago

There was that time it was bricking computers by erasing UEFI variables, but I'll allocate equal blame between systemd and UEFI

setquk2• 7 years ago

I’ve had Systemd completely stop responding before on numerous occasions on centos 7. As in can’t reboot or hangs rebooting or all commands hang.

Only recourse has been to reboot the instance from AWS dashboard.

I can’t get to the bottom of it because the tools don’t work when it’s down and there’s nothing there when it comes back up. I am not enjoying boiling to death in this pot of shit.

And then there’s the situation where it just won’t boot. I just fire up a new instance then because it’s easier than debugging it.

dvfjsdhgfv4• 7 years ago

Have you ever had pid 1 (systemd or any other init) crash? For the last ~three years I've been paid to maintain high reliability algorithmic trading systems that ran systemd and a whole lot of other stuff, and systemd has never crashed on me.

You see, that's the argument I hear a lot from Systemd advocates. The problem with anecdotal evidence is obvious. When you hear people opposing Systemd, practically all of them have some real-life issues with it, often related to functionality that would otherwise be non-essential (i.e. doesn't really need to be handled by PID 1). Of course if you don't have a particular problem, you don't feel it's important. That's precisely the attitude people resent.

nonbirithm• 7 years ago

Very recently I had this issue[0] as the result of a systemd upgrade, requiring the use of a recovery disk to downgrade to the previous version as the keyboard input had failed to be initialized.

[0] https://github.com/systemd/systemd/issues/11314

RX14• 7 years ago

Using systemd automount and NFS you can easily get pid1 unresponsive, hung in uninteruptible sleep forever.

stewartm• 7 years ago

Yep.

3AM, deep slumber, called out to look at a stricken server. Its problems included that systemd was frozen. Reluctantly I came to the conclusion that a restart was the only route forward. Cept, that is when you discover that the commands that have served you well for 2 decades don't work, as they are all wrappers for systemd, which has keeled over.

To this day, the `shutdown` man page, which I was checking in, makes no mention of how to resolve, tho in fairness the other commands (poweroff, halt, init) do. I discovered this after stumbling across https://github.com/systemd/systemd/issues/3282

If you find yourself stuck in the middle of the night, reading through docs to try and figure out how recover a machine with a crashed systemd, then `systemctl reboot -ff` or equivalent is what you are now looking for, the `-ff` being the key to "JUST £&*(ing RESTART THE MACHINE!!!".

Experiences like that, don't win you friends.

JdeBP12• 7 years ago

No. Not mine. Not systemd. Not others. And I touched upon how rare this was in practice in my experience some years ago on Hacker News.

* https://news.ycombinator.com/item?id=8384251

But it does happen to other people.

* https://unix.stackexchange.com/questions/440229/

And there was one crash that made the headlines.

* https://news.ycombinator.com/item?id=12600413

andrewstuart• 7 years ago

I really like systemd.

It's got tons of really great functionality.

systemd feels like it is well thought out and modern, like it's an integrated rethink and rebuild of lots of various software utilities that have evolved over many years.

I'm investing all my learning in systemd where appropriate instead of things I used previously like cron and supervisor.

CaliforniaKarl5• 7 years ago

I agree with that. But at the same time, systemd makes assumptions that I don’t think it should.

Case in point: I deal with several clusters that use Stanford Central LDAP for account info, and our UID numbers have gotten pretty high. So much so that it overlaps the range systemd uses for dynamic service UIDs.

The biggest annoyance is that the UID range is hard-coded, so I can’t provide an alternate range.

More details: https://github.com/systemd/systemd/issues/9843 (but please don’t spam the GitHub issue!)

tyingq4• 7 years ago

Wish my customers could file bugs and feature requests with that kind of thoughtful detail.

vinay_ys• 7 years ago

For an open source Free software project, I find it really strange pottering says this: "We don't recommend people to recompile their RPMs, the same way as we don't recommend changing the UID ranges..."

https://github.com/systemd/systemd/issues/9843#issuecomment-...

inferiorhuman25• 7 years ago

Do you remember when systemd would take usernames it couldn't parse (due to undocumented restrictions on valid usernames) in unit files and just translate them to uid 0?

darkpuma6• 7 years ago

"I'm not going to defend Lennart's approach to community interaction, but what I will say is you have to admire the willpower and determination of somebody who at work one day says 'I'm gonna write my own init system- actually no it's going to be a full on system layer for linux' [...] and then he gets it put into pretty much every major linux distribution."

Yeah, I suppose I also admire the willpower and determination of Genghis Khan.

Corbet is almost as dismissive as Lennart himself of the life-disruptive bugs real people encounter due to Lennart's approach to software engineering and project management. "It has bugs? It's software" This attitude is positively correlated with the amount of hate systemd (and pulseaudio) have gotten over the years.

HarryHirsch2• 7 years ago

If planes fell out of the sky due to buggy software as often as systemd crashes due to yet another bug, air travel would get much hate, too.

freddie_mercury• 7 years ago

If systemd cost as much as a plane, it would have fewer bugs.

We can go back and forth with dumb comparisons all day but I don't really think we're advancing the conversation.

darkpuma6• 7 years ago

Exactly.

Generally speaking people like things that work well and hate things that don't. Any other justification citing 'philosophies' or similar nonsense are retroactive attempts to cast their feelings in a more intellectual light.

When somebody says "I dislike systemd because it violates the Unix philosophy" what they almost certainly actually mean is "systemd has caused me a lot of grief." You don't hear complaints about firefox or chromium violating the so called "Unix philosophy" because generally firefox or chromium will satisfy the overwhelming majority of users. That's why nobody uses Uzbl.

This is what systemd advocates (like the article's author) don't seem to get when they point out that systemd is a launchd ripoff so people who like launchd should like systemd as well. People like launchd because launchd doesn't cause them grief. People dislike systemd because systemd does. Design philosophies have jack shit to do with it.

Anyway the only real 'philosophy' Unix ever had was "worse is better" and systemd certainly seems to pay homage to that.

JdeBP12• 7 years ago

You have erroneously put Benno Rice's words into Jonathan Corbet's mouth.

shittyadmin2• 7 years ago

This article is a bit of a joke "It's software so of course it's buggy" isn't a great argument when you're replacing something that didn't suffer the same issues.

I just count my blessings that runit is widely packaged in every major distro because it can just happily sit on top of sysvinit, systemd, upstart, pretty much any init system and does things in a very simple shell script style, I really wasn't a fan of the weird ini-like format for systemd or several different tools I'm expected to learn just to read my (now binary) log files competently.

If you're sick of switching init systems constantly or don't want to have to write separate scripts for your linux box and your freebsd box even, I highly recommend checking runit out.

I'm sure I'll give it a serious shot eventually... in about 3 years once they work all the Poettering kinks out, just like PulseAudio. They're doing some cool things with cgroups and stuff, so I hope it gets there eventually.

cetra32• 7 years ago

I second using runit. We use runit to be able to use the same service definitions inside docker, on a VM or bare metal.

If you've ever tried to use systemd inside docker to bring up a couple of services, you would know the hoops you have to jump through to get it working.

(I understand that docker wasn't invented to run multiple services in the one container, but sometimes it can't be avoided and simplifies app deployment vastly I.e, using CI to test your service actually starts up as per its definition: just run up a quick docker image with runit and a service definition file)

pestaa• 7 years ago

I've only seen supervisord as the root process in multi-purpose containers. Is there any significant gain to using systemd instead?

antt2• 7 years ago

I'm sure I'll give it a serious shot eventually... in about 3 years once they work all the Poettering kinks out, just like PulseAudio.

Good luck. If you need anything more than "I play a three minute song" on Linux audio you need both some type of real time kernel and jack.

Lazare2• 7 years ago

When Lennart Poettering started thinking about the problem, he first looked at Upstart, which was an event-based system that was still based around scripts, but he concluded that he could do a better job.

I think I'd like systemd more if I had more confidence that he had done a better job.

I never really understood why Upstart didn't get more traction; was it just the Canonical backing made other distros avoid it, or did it have drawbacks I never ran across?

na853• 7 years ago

Indeed. Early systemd was a fucking dumpster fire and part of the reason I no longer use Linux.

danbolt• 7 years ago

What do you use nowadays? Do you feel it addresses things better?

djsumdog3• 7 years ago

The problem is full process management, and in that regard, upstart was pretty much still just scripts. I will say this, systemd tries to solve the problem of tracking all children/grandchildren which is difficult to do with sysvinit or scripts by themselves.

Systemd also solves the problem of unified ways to start services across many distributions. Packaging things is easier. I don't like the systemd ini target format and wish something else had won out instead. With something like upstart, in theory you could rewrite the upstart daemon and use the same scripts, since the format/standard was much simpler.

I feel like there must have been a lot of bullying to get systemd across so many systems. It just seems weird to see so many major distros all decide to accept something so complex if it wasn't for Redhat cramming it into everything they maintained and helped fund.

JoshTriplett3• 7 years ago

systemd tries to solve the problem of tracking all children/grandchildren which is difficult to do with sysvinit or scripts by themselves.
Systemd also solves the problem of unified ways to start services across many distributions. Packaging things is easier.
It just seems weird to see so many major distros all decide to accept something so complex

You answered your own question. Distributions adopted it because it solved many problems for them.

pas14• 7 years ago

I feel like there must have been a lot of bullying to get systemd across so many systems.

I followed the Debian discussions when they debated init changing.

It was not pretty, there was a lot of things, but I don't remember any bullying. It was simply a very long uphill battle from a sort of loose group against the majority of maintainers that wished to stop worrying about 99% of init script problems.

Systemd is not perfect, never was, but it gets shit done, whereas no other project does in this regard. (A lot of people wanted to tackle some parts of what systemd does, but that was late and insufficient.)

And, all in all, there is always room for a systemd2 in Rust, distros would switch to it in a heartbeat, if it would be better.

geofft6• 7 years ago

I used upstart deeply around 2012-2014 in a commercial product based on Ubuntu. I remember two problems with it which had me waiting for Ubuntu to switch to systemd (unfortunately my employer ran out of money right when that happened). The first was that Upstart's model is upside-down: where systemd has units that depend on units, Upstart has jobs that are activated as other jobs complete. Whereas you would tell systemd that (say) Apache depends on syslog and ask it to try to start Apache, and therefore systemd would first try to start syslog, you tell Upstart that Apache can be started when syslog starts, and that you should start syslog, and then Apache's dependencies are met and Apache can start too. Instead of having a single target (like sysvinit runlevels or systemd's multi-user.target) that it works towards, Upstart just starts ... stuff ... untill it's done. Upstart did have a runlevel concept, but as I recall it was an event that triggered jobs to start, not a target. See http://upstart.ubuntu.com/cookbook/#critique-of-dependency-b... for Upstart's defense of this approach.

(systemd is hardly unique in being dependency-based instead of event-based - but it is a strike against Upstart in particular. Also, while one of the stronger claims of why Upstart should work this way is it matches the evented mindset of udev, systemd can handle events from udev just fine, by just having it add additional targets - and in the end, systemd consumed udev.)

The other was that it was hard to deal with its bugs. I had a job that wasn't activating. I spent days chasing it, including attaching a debugger to Upstart, and eventually gave up. I've used systemd extensively and have not had "Why isn't this service starting" bugs. (I've certainly had that confusion, but I've always been able to figure it out quickly.) systemd has in my experience been reliable and trustworthy. I'm not going to say that it's bug-free, or that I like its implementation, or that I have no complaints about it. I am going to say I haven't been frustrated by it - even when running very old distro versions with lots of known bug fixes upstream. That's very important for whether I'm happy with it or whether I'm going to switch back to sysvinit and find something else for service monitoring.

Lazare2• 7 years ago

Hey thanks, that's good info and helps me understand the bigger picture. Very useful comment!

blihp2• 7 years ago

I think it largely was Canonical that was the issue. Every time they've pushed something that they designed/developed it always seemed to me that they were more attempting to solve their business model problem rather than a Linux architectural problem. (i.e. get people dependent on them rather than just providing a solution)

Even though Red Hat is a much larger organization, their contributions have never felt like power plays in the same way that Canonical's have. Perhaps it's just the way that Canonical tries to shove their stuff through while Red Hat seems to get more organic buy-in. For example, as controversial as Systemd was (mainly from sys admins and power users), it did have a level buy-in from distro maintainers and developers which Upstart never really did. Canonical just tried to say 'we're doing this.'

inferiorhuman25• 7 years ago

Even though Red Hat is a much larger organization, their contributions have never felt like power plays in the same way that Canonical's have.

Since when? I remember some particularly contentious things like EGCS and various kernel doodads, and of course systemd.

the_why_of_y4• 7 years ago

The answer is in this Google+ post:

https://web.archive.org/web/20140928104327/https://plus.goog...

   Scott James Remnant
   +
   4
   1
   2
   1
   Reply
    
   +Michael Hasselmann at the point that Kay, Lennart and
   I sat down and discussed all this stuff, I don't think
   Upstart was perceived as "shitty" at all. We'd had
   on/off discussions for ages, but the big one I remember
   was the LF Collab Summit in SF in April 2010.

   Hindsight certainly lends a different perspective, and
   I'd be the first person to say that Upstart doesn't
   work as intended. +Lennart Poettering makes a great
   point about mountall in a recent post, it was written
   because Upstart couldn't do the complex filesystem
   cases it was designed to be able to do; and I was very
   aware even at the time that was a failure that would
   need to be addressed.

   Had the CLA not been in place, the result of the LF
   Collab discussions would have almost certainly been
   contributions of patches from +Kay Sievers  and Lennart
   (after all, we'd all worked together on things like
   udev, and got along) that would have fixed all those
   design issues, etc.

   But the CLA prevented them from doing that (I won't
   sign the CLA myself, which is one reason I don't
   contribute since leaving Canonical - so I hold no
   grudges here), so history happened differently. After
   our April 2010 meeting, Lennart went away and wrote
   systemd, which was released in July 2010 if memory
   serves.

   So I don't think I can claim that the perceived
   shittiness of Upstart spawned systemd, because at the
   time it wasn't seen that way. I don't think I can even
   claim that it provoked Lennart in any way, init was an
   area all distributions were fiddling with, so it was
   inevitable anyway.

   I entirely agree with Kay and +Greg Kroah-Hartman  that
   it was the CLA that caused systemd to be written
   instead of Upstart.

   But I don't need that self-affirmation anyway :) I
   wrote Upstart, I got paid for it, I moved on to do
   other things, something else came along and replaced
   it. If Upstart hadn't been under the CLA, and systemd
   hadn't've happened, all my code would have long since
   been rewritten by now anyway.

   That's the nature of the software world, there's no
   point getting precious over things. Do your bit, have
   fun doing it, move on and let others do their bit,
   etc.

anfilt3• 7 years ago

Systemd just makes things worse.

I remeber seeing someone post one of the simplest init process one could write. It was only about 100 lines of C.

geofft6• 7 years ago

It's much less than that, but it's sort of an uninteresting demo I think - you could pretty easily write an init process that just spawns systemd in a container, for instance, and run the whole system in a container. Or write an init process that spawns systemd as pid 2, runs prctl(PR_SET_CHILD_SUBREAPER) to make pid 2 responsible for reaping instead of pid 1, and then sits around carefully doing nothing. If systemd crashes, whatever, hopefully sshd etc. are still up.

Of course, if you do that then the question is why even bother with an init process. Patch the kernel to not treat pid 1 specially. If a process gets orphaned, don't reparent it to init, just reparent it to nonexistent pid 0. Have the kernel deal with the thing that the 10 lines of C would be doing (waiting on processes to terminate).

I think there are meaningful complaints about systemd but the fact that it's a complicated pid 1 is not one of those. If it were actually a problem someone would have written and productionalized one of the above two approaches.

anfilt3• 7 years ago

It was code that even let one make simple files to start up one shot process or daemons. If a long running daemon died it would restart it. Plus it handled the getting the status of orphaned processes, and few other things.

Really the biggest issue with systemd is is scope creep scope creep means more and more packages will have systemd as a hard dependency. Like there some aspects of gnome that will not work without systemd, but at least it's not entirely a hard dependency just means you lose features unless you patch gnome.

djsumdog3• 7 years ago

I hate how he talks about knee-jerk reactions to change. because I don't think that's what's going on here. I remember I first saw systemd on a release of opensuse. I didn't think anything of it at first, except that I didn't really like the command line interface (systemctl) and I found the flags and options cumbersome. I often see new software in new releases (including the old HAL/DBUS layer) and didn't have the same reactions (although HAL had a lot of issues and was later removed or merged into dbus).

I've seen the BSD talk on this and I agree, having a system layer is helpful. It'd be nice if it was plugable, NetworkManager (or others that have some standard messages you can send/get via dbus), consolekit OR logind, etc.

systemd does make it nice that I only have to write startup/shutdown scripts once for each distro, but I'm not happy with the layout of target files, the way mounts are handled, some of the weird race conditions I've found between systemd mount targets and fstab, etc.

systemd is modular, but the modules are still all part of the whole and are not easily replaceable. The same can be said when Docker went to a modeler refactor, but there are alternative implementations of the entire docker engine. Every attempt to create alternative implementations of systemd have eventually gone unmaintained because systemd keeps getting more and more complex and engulfing more systems.

If it wasn't for distros like Void, Gentoo, Alpine, Slackware, et. al, we'd no longer have a choice at all. There would be some things that simply couldn't be deployed on embedded systems because all of the dbus shims just wouldn't exist.

It's not that people are opposed to change, it's that there are legit concerns about some of the ways systemd works and is implemented, and the way it's been ham-fisted as a political move in a lot of ways.

Honestly, I don't think it will matter in a few years. I think the way things are going, eventually all services will be hosted via docker containers and it will be much easier to make Linux distros that have a tiny init layer that just launches a docker daemon and services. RacherOS already does this, with the init process being a container, which can be uses to start up shell environment containers and other service containers.

lsc2• 7 years ago

It's not that people are opposed to change, it's that there are legit concerns about some of the ways systemd works and is implemented, and the way it's been ham-fisted as a political move in a lot of ways.

I personally think the industry needs a lot more resistance to change when it comes to interfaces and other things humans have to understand.

I mean, I'm not talking about systemd in particular; I'm talking about in general about how interfaces change over time and people don't seem to take into account the cognitive costs of that change. Sure, ss is better than netstat and IP is better than ifconfig... but how much of that 'better' could you have done in a way that didn't toss away the historical knowledge so many people have of those tools?

And really, sysadmin tools are the least of it; I mean, they are operated by professionals, so if you want to pay for retraining (or pay the costs associated with there being fewer of us)

People change customer facing interfaces to no benefit all the time, forcing people who are trying to do other things to put effort into re-learning their interface.

I mean, my point is that interface changes are expensive, and should not be undertaken without a really good argument that they bring more benefit than the cost of retraining.

pas14• 7 years ago

First we need sane (secure, semantic, programmatic) interfaces that slowly become standards.

netstat? /proc files? ss? parsing text? wtf?

I mean, sure why not, but at least don't call them interfaces. they are userland apps people like to script, because they are lazy to use libnetlink (or libwhatever thay uses the right kernel interface, if it exists at all).

That said, the recent gmail ui change made me reconsider Thunderbird again. And android looks different every year. sometimes it's better, sometimes it's worse. iptables, nftables. http1, http2 (and now 3 over UDP). change is the only constant.

geggam2• 7 years ago

Not everything runs well in a container. In fact with the lack of network understanding in most container implementations I would say there are many issues.

Additionally this concept of stateless container design and state kept in containers there are opposing implementations

etaerc• 7 years ago

docker is currently killed by the kubernetes community btw. It's a slow death in some regards (1+ years) but quick in regards to "it will replace anything in 10 years".

JdeBP12• 7 years ago

systemd is modular, but the modules are still all part of the whole

The idea that you are searching for is coupling. Modular systems should aim to have low coupling and high cohesion.

e12e• 7 years ago

You know, there's so many "big" wrong decisions with systemd design and assumptions (in particular, there seems to an unhealthy focus on graphical desktops rather than headless multi-user servers, as this default of sigkill all the things on ssh connection drop default is but one example. And the monolithic design (DNS in the init?)).

But that said, does anyone know where on earth they came up with the command line ux? Like the names of the commands , and the parameters? I mean, they are like an April fool's joke...

mantlepro2• 7 years ago

A well-written, thoughtful article, worth the read.

Nerds have a complicated relationship to change; it's awesome when we are the ones creating the change, but it's untrustworthy when it comes from outside.

mantlepro2• 7 years ago

Yikes! How in the world did I lose 2 of my 3 karma points on this?! Somebody please explain and upvote? Linux user since the 90s but totally new to Hacker News. I'm good with constructive criticism, opinions, etc. but not a fan of downvotes without explanation.

stevefan19992• 7 years ago

I reckon there's a SystemD bug that didn't mount LVM volumes correctly, some has lost data that's real tragedy

JoshTriplett3• 7 years ago

Surely if there's an issue that serious, you have a bug report number?

shakna6• 7 years ago

Here you go: [0], from 2017.

[0] https://github.com/systemd/systemd/issues/6066

voltagex_• 7 years ago

I'd recommend everyone watch the video: https://www.youtube.com/watch?v=o_AIw9bGogo

dingo_bat• 7 years ago

My only issue is the binary log files. When you're writing out the logs to disk, just make sure it is in text! How hard can it be?? It creates unnecessary friction when dealing with logs from a crashed system.

XorNot• 7 years ago

Does this ever actually come up for all the people complaining about it? Where are these people working that they're not willing to install rsyslog which reads journald logs directly and will write plaintext, but who have this problem enough to warrant all the complaining?

Like, previously to have any logs at all you had to have a syslog daemon. This is not a new situation.

inferiorhuman25• 7 years ago

Like, previously to have any logs at all you had to have a syslog daemon. This is not a new situation.

What's new is that now more and more-difficult-to-debug pieces are required. For what benefit?

yarrel2• 7 years ago

This is a "now you have two problems" solution.

Karrot_Kream• 7 years ago

We specifically use runit because we've had systemd issues in the past. Systemd may have matured now but at this point we're moving over to containers anyway, so it doesn't matter.

makz• 7 years ago

As a conspiracy theorist, I'm going to say that systemd is aimed towards destroying Linux and everything it represents.

They want to push the cloud, to better control us, take away power from us, leave us at their mercy.

The next step is, when everything that matters is on the cloud, they are going to replace Linux with something else that only they know how it works on the inside, and that's going to be the end of it, we'll be left only with paywalled APIs and services, but they'll say it's better for everyone and all of that. It's sad.

newnewpdro4• 7 years ago

You're confusing systemd with docker and kubernetes.

busterarm2• 7 years ago

I hate how accurate this is. The more I work on Kubernetes implementations, the more I feel like it's a tool that 99% of companies can't effectively operate. The attitude is very reminiscent of the mainframe era where ops didn't want mere users touching their precious systems.

I feel like as an industry we're wasting tons of cycles solving the wrong problem.

squirrelicus• 7 years ago

I hate how the Unix philosophy argument is boiled down to "one vs many" binaries. It's about independence to the degree of reusability. There's nothing in systemd suite that's useful unless you're two feet in. You can say things about streams and text and binary formats, but those are distractions. Systemd tooling is tightly coupled to each other. It's a huge codebase, a huge system that shoddily replaces that which was historically done with less. And it's the biggest reason I don't run a Linux workstation anymore.

progman2• 7 years ago

Basically said, systemd follows quite the opposite of the KISS principle which made Unix and Linux so great.

jiqiren• 7 years ago

My impression of systemd: shoddy version of launchd.

myrandomcomment• 7 years ago

I hate systemd with a passion, however I understand the reason for it. I think that systemd is a bit Franklinstien by the nature of having to pull udev and d-bus into it. So I guess it is a victim of having to deal with the system it is on. Oh well.

I think the only Unix system to get this right was likely Solaris via SMF.

https://en.m.wikipedia.org/wiki/Service_Management_Facility

WindowsNT also did it correctly.

j16sdiz• 7 years ago

I hate parallel service start up on server.

Yes, it is faster. But when it breaks because bad dependency, it is hard to reason the breakage on a initrd shell on a small remote console.

I value debuggability, reliablity and manual fallback options on server.

kakwa_• 7 years ago

I hate writing systemd .service files.

I'm always confused with the Type (should it be simple? forking?) and my program ends-up starting but not stopping or things like that.

I know I lack the knowledge to write a good .service file in one go, as I write one every other month at best and forget all the details in between. It would be the same with a traditional init scripts. However, with init scripts, the way to debug is obvious: just add a '-x' somewhere and here we go, you have a detailed execution. I don't know the equivalent with systemd and I end-up bashing my head against the wall trying everything blindly.

throw2016• 7 years ago

The author goes into 'contempt culture' which is valid in any group of people ie 'PHP haters' but then tries to slip through the old trope of 'resistance to change' which is a clever bait and switch that positions change as 'always good' and refashions dissenters as anti-change which is a time tested PR strategy to push through unpopular changes that benefit a few by positioning any 'dissent' as 'anti change' and thus avoiding the meat of the discussion - the change in question.

We can agree on the problem and then on the solution, but any proposed solution can't just pass through 'without scrutiny' because a problem exists.

Systemd proponents since the beginning have positioned any criticism as 'motivated', 'greybeards', 'haters' or anti-change which tells you that there is no space for discussion here, only 'acceptance'. Can there be any legitimate criticism of systemd without this kind of politics, hypersensitivity and now posing as victims?

aidenn02• 7 years ago

Also the whole "One solution for people who don't like it is to create their own alternative; that is a good way to find out just how much fun that task is."

This seems to imply that those criticizing it didn't have alternatives, which is just not true. However it is so monolithic that as soon as a tiny part is depended on by something else (e.g. gnome) alternatives quickly become a non-starter.

systemd seems to be getting rewarded by not playing well with others which is something that plays strongly into a lot of people's dislike with it.

jimrandomh6• 7 years ago

Back in 2016, systemd started killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen). This caused problems for programs like nohup, screen and tmux, which deliberately keep running. Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API. As far as I know, none of them did.

Two years later, you can find hundreds of support requests across the internet, from frustrated users who are having their sessions killed by systemd.

Bugs are annoying, but that's life. On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating. I don't think systemd's developers deserve the trust that maintaining a core piece of infrastructure requires; they don't seem to care enough about whether they've broken things.

MereInterest4• 7 years ago

Absolutely. I can understand implementing this feature for some special cases, like containers that should clear all hint of a user away on log off. It should never have been the default, and breaks an entire category of software. In my standard .bashrc file, I have the following snippet to warn me if I am on a system with that stupid setting enabled.

    if which loginctl > /dev/null && loginctl >& /dev/null; then
        if loginctl show-user | grep KillUserProcesses | grep -q yes; then
            echo "systemd is set to kill user processes on logoff"
            echo "This will break screen, tmux, emacs --daemon, nohup, etc"
            echo "Tell the sysadmin to set KillUserProcesses=no in /etc/systemd/login.conf"
        fi
    fi

m45t3r4• 7 years ago

Thanks, now I know why Emacs daemon keeps delaying my restarts in the system (just discovered that NixOS defaults KillUserProcesses to false).

Turning this on to true, for me it does no make sense to a user service (yeah, I run emacs as a user's systemd service) to keep running after I logout of my system.

P.S.: And the fact that for some people this behavior makes sense is why I think Lenart decision to put this as an option makes sense.

newnewpdro4• 7 years ago

There is no NOHUP signal, you're referring to SIGHUP.

See the enable-linger option for loginctl and KillUserProcesses for logind.conf. KillUserProcesses was set to default enabled on 4/9/2016, prior to that it didn't happen, but was configurable if desired. So you were always able to change the config to restore the previous behavior from the moment the default turned it on.

Edit:

Here is the commit where it happened

https://github.com/systemd/systemd/commit/97e5530cf2076a2b4f...

mirimir7• 7 years ago

I just checked a few Debian stretch boxes that I setup, and "KillUserProcesses=no" is set on them all. And until a few minutes ago, I didn't even know to check.

So how can it be the default?

jimrandomh6• 7 years ago

I meant SIGHUP. Edited.

JdeBP12• 7 years ago

So you were always able to change the config to restore the previous behavior from the moment the default turned it on.

No, you were not.

The thing that people are missing here is that neither of the systemd-logind behaviours, with KillUserProcesses=yes or KillUserProcesses=no, is the long-standing behaviour of kernel login sessions all of the way back to 7th Edition that nohup, tmux, screen, emacs --daemon, mosh-server, deluged, and more all interoperate with.

The behaviour of kernel login sessions is that end of login session is a HUP signal to the session leader, and that termination of the entire TTY login service (such as at system shutdown) is a TERM signal to everything followed by a KILL signal to everything then remaining.

The systemd-logind session behaviour with KillUserProcesses=no is no signals at all at the end of the login session, and at termination of the TTY login service both HUP and TERM signals together then KILL signals, to everything.

The systemd-logind session behaviour with KillUserProcesses=yes is both HUP and TERM signals together then KILL signals, to everything, both at login session termination and at TTY login service stop.

As I pointed out years ago, the fix is to make systemd-logind use KillUnit at hangup and StopUnit at service termination, actually providing the conventional behaviour which it currently does not in any mode and addressing the original problems (with some background GNOME utilities in a login session that were never being sent a HUP signal at logout and would have exited had they been) that motivated this whole mechanism in the first place.

* https://news.ycombinator.com/item?id=12335128

* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221

* https://news.ycombinator.com/item?id=11798604

xyzzyz5• 7 years ago

Systemd's response was to say that they should incorporate systemd's library, and use systemd's new daemonization API.

By "use systemd's new demonization API" you mean, instead of

$ screen

systemd asks you to write

$ systemd-run --scope --user screen

instead. Annoying to have to learn a new thing, but hardly the unbearable burden.

On the other hand, when you're an impacted user who's lost work, and researching the bug leads you to a years-old discussion in which someone is actively denying that the bug exists and refusing to fix it, that's infuriating.

Because it's a bug for some, and intended behavior for others. Look, you make it as if they introduced a bug on purpose to screw with some people. It's clearly not the case, there was a specific tradeoff involved.

nerdponx• 7 years ago

It's a bug because it violates the expectations of an uninformed user. You aren't given a warning about it, it's not documented in big bold letters anywhere, and it's also not POSIX compliant.

zorpner4• 7 years ago

Annoying to have to learn a new thing, but hardly the unbearable burden.

Rather, a breaking change to everyone's scripts and processes for zero benefit.

stiff• 7 years ago

They changed a decades-old behavior many people rely on, and it must have been obvious from the start people will loose work because of it.

cyphar3• 7 years ago

Annoying to have to learn a new thing, but hardly the unbearable burden.

The problem is now your scripts won't work on systems that don't use systemd. Shell scripts work on FreeBSD, but now you can't use them because they require systemd-specific code.

I am not necessarily anti-systemd in most respects (I like a declarative definitions of services and less shell script hell), but the fact that they keep trying to get people (including container runtime developers like myself) to use _their_ API rather than the preexisting ones is fairly "anti-social".

shakna6• 7 years ago

Because it's a bug for some, and intended behavior for others. Look, you make it as if they introduced a bug on purpose to screw with some people. It's clearly not the case, there was a specific tradeoff involved.

They broke userland.

It doesn't matter what tradeoff they made - they went against POSIX behaviour, and as a result, broke numerous utilities, both past and future.

Let's say that again - systemd introduced breaking behaviour on userland, against POSIX, and instead of backing down and allowing for expected and specified behaviour, they said it's everyone else's problem.

That is neither professional, nor responsible.

When you make a mistake, a mistake that breaks the behaviour of POSIX, and POSIX utilities like _cron_, you apologise, and fix the problem.

You don't turn around and say that all the sysutils should incorporate your new idea.

MertsA10• 7 years ago

That comes down entirely to how systemd is configured. If you don't like what your chosen distro has picked as the default then complain to them. systemd didn't force anyone's hand on the subject, they just added the feature. It's a pretty natural design choice IMHO. When I want to log out, I don't want to let some hung up daemon keep running just because it wasn't able to process the SIGHUP sent to it.

How else do you propose to make sure that when I log off my ssh-agent is really terminated and not just locked up with my keys still in memory? The POSIX approach is insufficient, there's no way to know if a process received a signal and chose to ignore it and keep running or if it received a signal but it was deadlocked and kept running.

inferiorhuman25• 7 years ago

How else do you propose to make sure that when I log off my ssh-agent is really terminated and not just locked up with my keys still in memory?

Perhaps with a signal handler?

semi-extrinsic• 7 years ago

If only there was a way for the system init program to identify and keep a list of processes it has spawned, you could imagine like a unique numerical Process ID, and then if there was a program that could check the Process Status, and another that could kill the process identified by this... PID with increasing levels of aggressiveness...

zrm8• 7 years ago

The problem is that you're breaking compatibility by changing the default. It's one thing to add a feature that can solve a problem. It's something else to break existing programs that don't use it.

If you're not going evaluate each individual program to determine whether the new behavior is appropriate then it should be opt-in rather than opt-out. Then ssh-agent and anything else that knows it should be forcefully killed can opt-in without breaking other innocent programs.

wbl2• 7 years ago

If a process doesn't handle SIGHUP it dies. So all the daemon has to do in that case is nothing.

jimrandomh6• 7 years ago

Systemd developers, if you're reading this: this isn't the sort of bug where people grumble for awhile and then get over it, because things are still broken, and the workaround being circulated (KillUserProcesses=no) doesn't fully work. (https://github.com/systemd/systemd/issues/8486) As long as people continue to encounter this issue anew--and they still are--people will be angry at the systemd maintainers.

avar• 7 years ago

The bug you've linked to was closed[1] by the reporter with "Thanks for the clarification guys. Much appreciated!", after it was pointed out to them that something they were trying ho do with "KillUserProcesses=no" was better done in another way.

1. Edit: Not literally closed by the reporter. Lennart Poettering closed it, "closed by the reporter" as in "the issue was resolved to the reporter's satisfaction".

jesuslop• 7 years ago

Is there a daemonization API as such? I think there was only the "way of doing" shown in man 7 daemon.

JdeBP12• 7 years ago

The systemd people have their own version of that manual page.

* https://freedesktop.org/software/systemd/man/daemon.html

IBM was explaining what to do back in 1995.

* http://jdebp.eu./FGA/unix-daemon-design-mistakes-to-avoid.ht...

masonic• 7 years ago

  killing user processes on logout

By "killing", do you mean some other signal than (or in addition to) SIGHUP? Does it send SIGKILL?

seba_dos1• 7 years ago

That's the whole issue here. It does.

poettering9• 7 years ago

You know, because we knew this would be controversial we made sure it was both a compile-time option and a runtime option. Yes the upstream default of both defaults to on, but that's just upstream. We made it very easy and supported for downstream distros to switch between opt-out and opt-in of this option for their users. We have encouraged distributions to leave it on, but we were fully aware that for compatibility reasons this is something downstreams likely wanted to turn off, and most compat-minded distros did, as we expected.

Now I am used to taking blame for apparently everything that every went wrong on Linux, but you might as well blame your downstream distros for this as much you want to blame us upstream about this, as it's up to them to pick the right compile-time options matching their userbase and requirements in compatibility, and if they didn't do that to your liking, then maybe you should complain to them first.

(And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite". I mean, that's not how you build a secure system. We fixed that really, fully knowing it would depart from UNIX tradition, but that's why we made it both compile-time and runtime configurable)

(Also, nobody has to "incorporate" systemd's library to avoid the automatic clean-up. In fact, there's no library we provide that could do that. What was requested though is to either run things as child of systemd --user or just register a separate PAM session, neither of which requires any systemd-specific library.)

Lennart

tasuki• 7 years ago

Lennart, thanks for the information. Mind explaining why you chose to kill user processes on logout as the default?

majewsky7• 7 years ago

What was requested though is to either run things as child of systemd --user or just register a separate PAM session

When you run your screen or tmux below `systemd --user`, you still would have to `loginctl enable-linger`, no? I remember having to do that when I set up a PulseAudio server on a headless machine where I don't maintain an active session.

vesak• 7 years ago

You've done great work as a whole, as you probably know. Try not let the lowlifes get to you.

inferiorhuman25• 7 years ago

Now I am used to taking blame for apparently everything that every went wrong on Linux, but you might as well blame your downstream distros for this as much you want to blame us upstream about this, as it's up to them to pick the right compile-time options matching their userbase and requirements in compatibility, and if they didn't do that to your liking, then maybe you should complain to them first.

It's up to you as a systemd developer to pick sane defaults. Claiming that it's okay to introduce opt-out breaking changes upstream and then abdicate responsibility is a quite bit like walking around while waving your hands and arms around and then blaming whoever you hit for walking into you.

sfilargi2• 7 years ago

Completely agree. The problem is not upstream, but downstream. Distros should have done better job and chosen a better default system manager and not systemd.

You build your software the way you want and like. If others don’t like that it breaks POSIX they should stop using it instead of complaining. Or fork it.

inferiorhuman25• 7 years ago

And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite". I mean, that's not how you build a secure system.

As an aside this is the height of arrogance to suggest that the systemd is somehow a more secure alternative. Lest this be considered an empty ad hominem attack, let me quote the pwnie you won in 2017[1]:

Where you are dereferencing null pointers, or writing out
of bounds, or not supporting fully qualified domain names,
or giving root privileges to any user whose name begins with
a number, there's no chance that the CVE number will
referenced in either the change log or the commit message.
But CVEs aren't really our currency any more, and only the
lamest of vendors gets a Pwnie!

1: https://pwnies.com/archive/2017/winners/#lamestvendor

sametmax• 7 years ago

For what it's worth, systemd makes my life easier.

When I switch distro, it's almost always systemd, and not the system du jour, so I know how it works. Creating service files is a google query away, and makes common use cases a breathe, while advanced features that were hard to bash script yourself into, are now just a few options to type.

I understand that many people may have problems with systemd for their particular situation, but that's not my experience.

As a dumb user with a few laptops and servers that needs an occassional daemon, I'm glad systemd won. I know you get a lot of heat since it came out, so thank you for working on it.

Asooka• 7 years ago

So you made the default the worst possible option, because... why exactly? And now that the problem is apparent, you haven't changed the default because...? I don't know what goes through your and the rest of the systemd's team's heads, but good software engineering it is not.

loudtieblahblah2• 7 years ago

You know, because we knew this would be controversial we made sure it was both a compile-time option and a runtime option.

This is standard from you. You knock the glass on the floor and blame the maid service for not cleaning up after you.

It's everyone's faults but yours.

And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite".

Oh how hyperbolic. Nuances and caveats in terminology is not a weakness.

I don't see why you're splitting hairs over this but can't be bothered to care about your UID numbering bug.

Or he fact systemd-resolv is responsible for DNS leaking on VPNs.

But yes, tell me more about how a functionality that enables terminal multiplexes is a "weakness"

Now I am used to taking blame for apparently everything that every went wrong on Linux,

It's because of your smarmy, arrogance.

You break POSIX compliance, which has a real world effect in multiple areas and you accept bug reports with the humility of Donald Trump being interviewed by MSNBC.

Then when you retreat into your safe space, you play victim to the situation you created.

You talk of Linux culture toxicity, smearing the likes of Linus Torvalds, while essentially being the metaphorical sibling putting your finger in people's face repeating "I'm not touching you" over and over. Then you acted attacked when someone claps back.

You're a cry bully hiding behind a vaneer of professionalism acceptable for Red Hat's HR department which enables you to mark one more bug as "wontfix"; your attitude, your arrogance, your conceits that things not broken in fact, are so you can provide solutions no one asked for and no one benefits from.

gerbilly2• 7 years ago

I still consider it a weakness of UNIX that "logout" doesn't really mean "logout"

Ok, but UNIX and it's behaviour has evolved over forty years, and users have a certain set of expectations about it.

Also, it should be noted, systems like UNIX are cultural artifacts. The way they are is the result of forty years of back and forth debate and negotiation and eventually compromise.

I can't speak for all of them, but I think that people that are bothered by systemd are upset that all of history has been brushed aside to make place for the preferences of just a few influential developers.

Whether a feature like logout is "logical" or not, is besides the point. Operating system design isn't just about logic, it's about serving users.

jimrandomh6• 7 years ago

There's a bug here, which impacts end users: a variety of programs which are clearly intended to persist in the background (nohup, tmux, etc) are failing to persist. This is a real bug. We care about it. I won't be satisfied until it appears that the bug is on track to be fixed, and a lot of other people won't either.

The options for fixing the bug are:

* nohup, tmux, emacs, etc all take dependencies on systemd and use the new systemd daemonization procedure. This is not a viable path because the maintainers of those utilities have refused (see https://github.com/tmux/tmux/issues/428), and because there are too many of them.

* Each distro separately works around the problem by maintaining forks of nohup, tmux, etc. This is not a viable solution because it's way too many forks; people will be finding broken distro+utility pairs forever.

* Each distro separately works around the problem by putting loginctl enable-linger in /etc/profile and KillUserProcesses=no. This would effectively be overruling a systemd's decision. Some distros won't know they need to do this, and the github systemd repo becomes a trap.

* Or: systemd backs down and changes the defaults so that the old daemonization APIs work again.

If you have a fifth option, we'd all love to hear it. But the status quo is that there's a user-facing bug, and the bug is still there. Rather than make the case for it not being a bug, you're currently making the case for it being someone else's bug, but the "someone else" doesn't actually have the power to fix it. You are the only one with the power to fix this bug.

yellowapple2• 7 years ago

I feel like the kernel policy of "don't break userspace" would be a valuable one for y'all to adopt.

jjolla• 7 years ago

still consider it a weakness of UNIX that "logout" doesn't really mean "logout" ... I mean, that's not how you build a secure system

so, unix has been running for 20+ years laden with this security flaw? strange that nobody has been screaming out to plug it all this time.

this feels like you have a bee in your bonnet that it is not a very 'pure' logout by some interpretation of what a "logout" should be. imho, "logout" should mean what it has always meant in the past.

majewsky7• 7 years ago

killing user processes on logout (rather than send them the SIGHUP signal, as POSIX says should happen)

TIL what nohup(1) is for.

zbentley3• 7 years ago

Sort of. While it's debatable when SIGHUP should be sent as part of controlled system logout/whatever, the signal itself was originally used upon abrupt disconnection (hang up) of the controlling terminal of a program.

cout• 7 years ago

I agree that on Linux-based systems, SIGHUP is a reasonable mechanism for killing processes when a user closes an ssh session, and that ignoring SIGHUP is a reasonable way to avoid getting terminated.

I disagree that POSIX says that processes should expect a SIGHUP when a user logs out (SIGHUP means the controlling terminal was closed). I am not at all a POSIX expert, so please correct me if I misunderstand, but afaict POSIX explicitly does not specify what happens to the controlling terminal when a user logs out (http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_...):

POSIX.1 does not specify how controlling terminal access is affected by a user logging out (that is, by a controlling process terminating). 4.2 BSD uses the vhangup() function to prevent any access to the controlling terminal through file descriptors opened prior to logout. System V does not prevent controlling terminal access through file descriptors opened prior to logout (except for the case of the special file, /dev/tty). Some implementations choose to make processes immune from job control after logout (that is, such processes are always treated as if in the foreground); other implementations continue to enforce foreground/background checks after logout. Therefore, a Conforming POSIX.1 Application should not attempt to access the controlling terminal after logout since such access is unreliable. If an implementation chooses to deny access to a controlling terminal after its controlling process exits, POSIX.1 requires a certain type of behavior (see Controlling Terminal ).

cryptonector• 7 years ago

Eleven years earlier when SMF was added to what would eventually be Solaris 10, we had this same problem. Some of us had to drop everything to fix "bugs" in cron, sshd, ... introduced by SMF.

Systemd is basically SMF, done poorly, because NIH.

JdeBP12• 7 years ago

Also two years ago, I explained how one could make this work, by having logind use KillUnit at hangup and StopUnit at shutdown.

* https://news.ycombinator.com/item?id=12335128

* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221

m463• 7 years ago

I believe systemd solves a problem or two.

However, I believe the implementation is not great.

A couple specific observations:

* lots of binaries. /lib/systemd, the logs. This more than anything else is a tragedy. Binaries get in the way of viewing, diagnosing, understanding and changing things. It seems like premature optimization to me - I don't think speed requires it. I think the majority are completely unnecessary.

* copying from launchd? Commercial software vendors ship binaries. They are aligned with them. They resist disclosure, which protects from reverse engineering, lawsuits and user modifications. Does that align with linux?

* the config files are a disorganized mess. It is like /etc/init.d but poorly understood and then /etc/rc.d was lumped into the same directory and organization became chaos. I looked at one system here and in /etc/systemd there are 16 directories, 23 symbolic links and just 11 files. Try unraveling the interdependencies.

additionally the config files (which mimic windows config files) lack depth. Simple config files can have benefits, say if a GUI read and wrote them to change settings, but I don't think that applies here. What I noticed is that they cannot do anything, so additional logic requires an intermediate script or compiled binary elsewhere.

* it is responsible for too many functions. I bet it didn't solve so many in its first iteration, but sucked in more over time. It reminds me of the accounting program that grew to rule everything in Tron.

I wonder if these problems could be resolved. It's hard to remove complexity and add elegance later.

atombender• 7 years ago

Launchd has been open source (Apache license) pretty much from the beginning: https://wiki.freebsd.org/launchd

mixmastamyk2• 7 years ago

Systemd as the new MCP, interesting. “Sark, prepare that daemon for immediate deresolution.”

poettering9• 7 years ago

I kinda like how a good chunk of the community complains we are "too monolithic", and the other side of the community complains we have "too many binaries". We can't win with you guys, can we?

Lennart

michaelmrose3• 7 years ago

In what universe does the number of binaries have anything to do with how monolithic something is?

cuillevel3• 7 years ago

Not that it matters, but I'm really happy about systemd.

Sure I have to learn new things, sometimes stuff breaks, but that has always been the case. For me most problems were with upgrading Linux distributions.

Anyhow it's great to have less distribution specific stuff and less shell scripts. Thank you and all the other systemd devs.

pdkl952• 7 years ago

"too monolithic" ... "too many binaries"
We can't win

Juxtaposing two unrelated issues usually isn't a winning strategy.

rauhl3• 7 years ago

Multiple binaries can still be part of a monolithic system (just like multiple services can still be part of a monolithic system in the web services world). A well-archetected system has strong separation of concerns — and well-defined, reasonable interfaces — between components.

If I’m writing a piece of code and I start wanting to rewrite the world around it, that’s a sign that I’m probably doing the wrong thing.

In the case of systemd, you actually have a decent case that a lot of things in Linux and the rest of the POSIX world could stand to be improved. A good, clean, portable, systems-focused approach could tackle all those things head on.

na853• 7 years ago

You really don't understand the complaint?

It's that the plethora of binaries are too tightly coupled and are not at all modular or interchangeable, which is materially the same as being a monolith.

I think you are smart enough to know this, and are just feigning ignorance.

majewsky7• 7 years ago

Try unraveling the interdependencies.

These dependencies between services have been there the entire time. At least now they're clearly expressed.

pas14• 7 years ago

lots of binaries. /lib/systemd, the logs. This more than anything else is a tragedy. Binaries get in the way of viewing, diagnosing, understanding and changing things

would you prefer sh/python/perl/lua scripts?

some (the suckless group) view C and make && make install the perfect way to observe and change things. (I like config files and runtime configurability.)

mirimir7• 7 years ago

Damn. So I'm just a simple Debian user. Systemd showed up, so I learned as much as I needed. And in some ways, I found it easier to use than init scripts.

Now I've been using Docker. And there is no systemd in containers. But of course, why would you want that? But the problem is that many packages now depend on systemd. Or at least, useful features do. So I'm back to fighting with scripts. It's funny, no?

tomjakubowski• 7 years ago

Maybe you wouldn't want systemd, but a basic init system as PID 1 is wise for a Docker container. See https://github.com/Yelp/dumb-init

mirimir7• 7 years ago

Wow. Thanks.

I've been wondering about all those zombies.

inferiorhuman25• 7 years ago

And there is no systemd in containers. But of course, why would you want that?

I knew a guy who invented an orchestration system that involved shoving an init process into docker containers. Much sadness ensued.

mirimir7• 7 years ago

Are you referring to dumb-init?

Is there a better way to reap zombies?

andrewshadura4• 7 years ago

Why are you saying there's no systemd in containers? Systemd works perfectly both outside and inside containers, docker or any others.

mirimir7• 7 years ago

I'm sure it does, however it's supposed to.

What I mean is that systemctl isn't available in containers:

   root@foo:~# systemctl status sshd
   bash: systemctl: command not found

Kivutar• 7 years ago

Some years ago, I was developing a Linux distro for embedded systems. Some of these embedded systems come with old kernels, and you will never get a recent kernel for them.

Systemd is written in C and relies on kernel headers. Which means that they need to have a layer that deals with different versions of the kernel headers. I remember it was headers about network.

At the time I was working on this, I had to contribute to systemd to add support for my kernel version. Got a few patches accepted in their master. And things were working fine for a while.

Until they suddenly decided to drop <= 3.10 kernel support. Yep. Afterwhat I had to maintain patch sets for systemd, which is a great amount of job, given that my upstream was updating systemd quite often.

Now my good people, explain me how such a mess could have happen with a previous init based on shell?

CaliforniaKarl5• 7 years ago

I'm curious: Did systemd drop 3.10 support before the kernel developers stopped supporting it (in 2017, see http://lkml.iu.edu/hypermail/linux/kernel/1711.0/03167.html), or before that? If it was around the same time, then I could understand dropping support, since even the kernel developers had decided to move on.

jononor• 7 years ago

You could have stayed on an older systemd version? Preferably one used by a major distro, such that someone else might backport security fixes etc for you.

Anyway the root cause here is bad vendor BSPs being stuck on old Linux versions.

teddyh• 7 years ago

GNU libc 2.26 (released 2017-08-02) requires Linux 3.2 or newer. It’s not like systemd is alone in requiring recent kernels.

aidenn02• 7 years ago

One of my "favorite" corner cases caused by systemd was that if an NIS user tried to login to the system, it would cause my X session to restart:

1. They changed the defaults for systemd-logind to disallow network access

2. When systemd-logind attempts to send a message to the NIS server, it instead hangs (or has a timeout greater than the systemd watchdog anyways)

3. The watchdog timer causes all systemd-logind services to be restart, which then tears down all of the child processes. My X session is a child process.

RemarkableMan• 7 years ago

I just don’t have time to beta test an init system with more LoC than the 2.6 kernel plus GCC 4. For now Devuan is where we landed but moving entirely away from Linux as Redhat moves system d into the kernel is the overall plan.

pgtan• 7 years ago

systemd broke UNIX. Everything that worked before, now is a mine field. Last week I had a SLES virtual machine refusing to boot, no network, just emergency shell. The reason was a forgotten line in fstab (without nofail option of course), so systemd couldn't mount some file system. Coming from Gentoo, this is just unbelievable.

the_why_of_y4• 7 years ago

Systemd correctly implements the "noauto" and "nofail" keywords in fstab(5) that were added in 2008 in the way they have been documented since 2010: if neither of these is specified for a filesystem, then that file system will be mounted at boot time and a failed attempt to mount that filesystem implies a boot failure.

  Util-linux-ng 2.14 Release Notes (09-Jun-2008)
  ==============================================
  mount(8) supports new "nofail" mount option.

https://github.com/karelzak/util-linux/commit/abe3d704b6aeb6...

The fact that previous init scripts gladly ignored these failures does not reflect negatively on systemd but on those init scripts.

Since 2011, FreeBSD has the same options with the same behavior, except that "nofail" is called "failok".

https://svnweb.freebsd.org/base?view=revision&revision=22283...

  Add a special mount option "failok" to indicate that the administrator wants
  the system to proceed to boot without bailing out into single user mode,
  even when the file system can not be successfully mounted.

By your argument, FreeBSD broke UNIX.

zlynx• 7 years ago

systemd refusing to boot without required drives is actually a favorite feature of mine.

Ever since I had an RH AS server boot without /var mounted. It was truly lovely with a gigabyte of trash filling up / then trying to decide if it was worth merging data or just throwing it out.

pdkl952• 7 years ago

This argument, he said, seems to be predicated on the notion that systemd is a single, monolithic binary.

Can we please stop misrepresenting the complaints against systemd? The only time I ever hear this "monolithic binary" argument is from systemd advocates. The actual complaint is about tightly coupling important features together. Not only does this make it difficult (often impossible) to replace individual components, when tight coupling happens at the (internal) protocol level, any replacement component ne4cessarily hast to implement a bunch of (sometimes unwanted) systemd baggage.

Busybox implements all of its features in single monolithic binary, but it isn't a monolithic design that tightly couples those specific components together. Replacing one of busybox's components is often as simple as removing busybox's symlink and installing the replacement. This isn't even a "Unix philosophy" issue. Even inexperienced designers shouldn't have as hard time Understanding why systemd is a monolithic design but busybox isn't.

chris_wot3• 7 years ago

What baggage are you specifically referring to?

lmm2• 7 years ago

It runs its own logging system with non-standard interfaces and formats. It runs its own DNS resolver with non-standard behaviour. It maintains compatibility only with a narrow range of udev versions, which in turn maintain compatibility only with a narrow range of kernel versions. And all the d-bus interfaces between these pieces may change at any point without notice. So you can't replace any piece of it, because even if you provide your own component that implements one of the systemd d-bus interfaces, you've got no forward compatibility.

duckerude• 7 years ago

https://suckless.org/sucks/systemd/ has items like "pid 1 does DNS". It's an incorrect complaint that exists in the wild, though it certainly isn't the basis of all accusations that it violates the Unix philosophy.

JdeBP12• 7 years ago

Thank you.

* https://news.ycombinator.com/item?id=13390245

ChuckMcM• 7 years ago

From the article -- Windows NT had a strong service model from the beginning, he said, and Mac OS has one now in the form of launchd. Other systems had to play a catch-up game to get there.

Reminds me of the saying, "and if your friends all jumped off a cliff, would you jump off too?" which exasperated parents use to council their children that emulating others needs to be done responsibly.

darkpuma6• 7 years ago

There is a big difference here. To continue with the cliff analogy: Launchd ran fast enough to clear the cliff and was rewarded by a refreshing dive into the cool water; it was implemented in a competent manner and consequently didn't cause people to become upset. Systemd on the other-hand can't run without stumbling, it hobbled off the cliff and bounced off every rock on the way down. It made lots of people pissed off because it interfered with their routines.

If systemd didn't cause problems for people, virtually no user would even know what it was, and consequently, virtually no user would hate it. If you ask your average OSX user what launchd is, they'll come up with a blank because launchd isn't causing trouble for them. If you ask your average linux desktop user what systemd is, a large portion of them will recognize the name, and many of them will chew your ear off out of frustration.

At the end of the day it's not about "adherence to philosophies" or any similarly dubious concepts. Whether or not software is hated is ultimately determined by whether or not it causes problems for people.

JdeBP12• 7 years ago

That's not the problem. Windows NT did have that right from the beginning. The problem is the second sentence, because it ignores that so too did Unix from the same timeframe. AT&T System 5 Release 4 had the Service Access Facility from 1988. AIX had the System Resource Controller from version 3.2. Other systems were not playing catch-up. They had this stuff too, at around the same time.

Benno Rice's explanation is actually ahistorical here. What actually happened is that the world spent a lot of time "cloning" existing Unix softwares during that time period, for reasons that we all known, and a lot of the time the clones were behind the times and did not catch up. By the time that Miquel van Smoorenburg cloned AT&T Unix init and rc for Minix, for example, what xe was cloning had already become years out of date.

* http://jdebp.eu./FGA/inittab-getty-is-history.html

chroma• 7 years ago

It is my opinion that Lennart Poettering is the software equivalent of Thomas Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.[1]

This is the person behind PulseAudio, Avahi, and systemd. In any sane world, these software projects would have been stillborn. Sensible alternatives would have outcompeted them. Instead, Red Hat did their best to foist them upon the community. They promised that after the initial adoption, all the wrinkles would be ironed out. All three times this was not true.

I hate to appear so cynical, and I'm certain that Red Hat didn't intentionally pursue this strategy, but it sure is convenient for them that the harder Linux is to use, the more they can sell support contracts. Not so convenient for people who actually want to use Linux.

1. https://en.wikipedia.org/wiki/Thomas_Midgley_Jr.

darkpuma6• 7 years ago

You might be giving Lennart too much credit. As the author of the article points out, Lennart didn't invent PulseAudio and systemd so much as he ripped them off (poorly) from coreaudio and launchd. To compare him to any inventor seems too flattering.

mixmastamyk2• 7 years ago

Also not a huge fan but it works better than a bunch of shell scripts.

redwall_hp• 7 years ago

Avahi is just a reimplementation of something Apple developed, too. Apple developed zeroconf networking, with their implementation of the standard being Bonjour. https://en.wikipedia.org/wiki/Avahi_(software)#Avahi_vs._Bon...

ernst_klim2• 7 years ago

Sensible alternatives would have outcompeted them.

Of course, and they didn't, because there were no sensible alternatives on linux. All these projects were clones of decently engineered macos projects, before they were created, linux people used some truly silly staff like sysvinit.

I would use something better than pulse and systemd, but there is nothing better available.

antt2• 7 years ago

Jack and runit.

They manage better performance in real world use cases with a fraction of the manpower invested in them. I wonder what will happen to the suite of projects supported by redhat now that they are owned by IBM however.

jiveturkey2• 7 years ago

All these projects were clones of decently engineered macos projects

systemd is the third world copy of Solaris SMF, is it not?

majewsky7• 7 years ago

It is my opinion that Lennart Poettering is the software equivalent of Thomas Midgley Jr., the inventor of both leaded gasoline and chlorofluorocarbons.

Considering that Midgley Jr. is (indirectly) responsible for the suffering and potentially death of many humans and other beings, I submit that this (in spirit) triggers Godwin's Law. The debate is therefore over.

zeveb3• 7 years ago

Considering that Midgley Jr. is (indirectly) responsible for the suffering and potentially death of many humans and other beings

Poettering is directly responsible for the suffering of many human beings, as indicated by threads like this. It'd be difficult and unfair to accuse him of being even indirectly responsible for anyone's death, though.

Sammi• 7 years ago

Which alternative to PulseAudio, Avahi, and Systemd exists for Linux that provide the same features?

I'm not aware of any audio daemon that is as feature rich and flexible and PulseAudio. None of the alternatives fit the same general purpose use case that is expected in a modern desktop that needs to compete with Mac OS and Windows 10.

bjourne3• 7 years ago

If you use a modern Linux system with KDE, you can try removing PA. Odds are you won't notice that it is gone. Not saying that network audio is a bad feature per se, but in my ten years of using PA, I have never needed it. IMO, systemd and Avahi are fundamentally different from PA in that most users have no need for the latter but almost everyone for the former.

asark• 7 years ago

This is the person behind PulseAudio, Avahi, and systemd.

Huh, I didn't know he was behind Avahi. That... explains some things. Thanks.

fmajid• 7 years ago

Farce, not tragedy

yarrel2• 7 years ago

First as tragedy...

_pmf_• 7 years ago

If the current state is the tragedy, I'm terrified of the farce.

My favorite systemdisms:

* (already mentioned several times above; sorry) even detached and nohup'd processes will be killed if the systemd context that transitively led to the creation of the process is killed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394

* mounts being in a separate "mount" namespace that is not accessible normally: http://www.volkerschatz.com/unix/advmount.html

ComputerGuru• 7 years ago

For all those lamenting the death of standards, change for the sake of change, system breakage, bloat, and lack of user control: give FreeBSD a try (or any BSD, really).

progman2• 7 years ago

I like to share the following links to those who also don't like systemd.

http://without-systemd.org

Devuan (Debian without systemd):

http://devuan.org

More Linux Distros without systemd:

http://without-systemd.org/wiki/index.php/Linux_distribution...

fanf2• 7 years ago

Debian Stretch works nicely for me without systemd

arminiusreturns3• 7 years ago

I particularly like how it ripped out the old logging system but completely failed to replace it properly. Local might be ok, but shipping to a central log server? Just save yourself the headache and use syslog or some other shipper like filebeat, etc. How can something that can't even do logging properly suddenly become the defacto standard?

newnewpdro4• 7 years ago

I still think NSA used RedHat to push systemd for potentially nefarious reasons.

Wow, just wow. Do you have any idea how insulting this is to the folks dedicating their lives to writing this GPL software?

You should try having an actual conversation with the authors some time. These are real people and long-time GNU/Linux users and advocates you're implicitly accusing of being NSA collaborators.

arminiusreturns3• 7 years ago

RedHats collaboration with NSA is well documented. I think given what we know of past actions it's at best extremely naive to not at least entertain it as a possibility.

Linus himself semi-openly admits he's been approached...

https://www.youtube.com/watch?v=7gRsgkdfYJ8

CarlFK• 7 years ago

Notable quotes

  22:10 "Unix is dead."
  25:28 "Change is awesome when we are the ones doing it.
  27:20 "No one needed to send death threats over a piece of software."
  27:30 "Contempt isn't cool."

thekingofh• 7 years ago

Can I easily replace systemd on a distro that came with it? I wouldn't consider myself a Linux expert, but I got to a point where I could get around and started to understand the different init runlevels and how to set them, then everything was ripped out from under me and I no longer understand how to work everything anymore. Perhaps Linux boots faster now, I don't know, but it was really very frustrating I couldn't find a major user-friendly distro that worked the way I was used to.

JdeBP12• 7 years ago

The writing was on the wall for run-levels for a long time. Longer than Linux has existed, in fact.

* http://jdebp.eu./FGA/run-levels-are-history.html

gerbilly2• 7 years ago

There are parallels with city planning.

systemd reminds me of the efforts in some cities at clearing 'slums' to replace them with 'modern' apartment blocks.[1]

sytemd seems like a similar technocratic, modernist effort aimed at UNIX.

Operating systems are kind of like cities. One the surface they are often inefficient and messy, and people often wish they could come in and make them logical and rational, but people live in those cities and neighbourhoods, and sometimes the postbox is where it is for a reason.

If Jane Jacobs[2] were around, she could write a book called: "The Death and Life of Great American Operating Systems"

[1] See: https://en.wikipedia.org/wiki/Pruitt%E2%80%93Igoe, https://en.wikipedia.org/wiki/Quarry_Hill,_Leeds, https://www.youtube.com/watch?time_continue=862&v=Dxr2tEGlWU...

[2] https://www.citymetric.com/fabric/against-modernist-nightmar...

dekhn• 7 years ago

I've used linux systems since '93. I waited until the last minute to adopt systemd. So far, for me at least, it's been great. It's a lot easier to express service dependencies, find problems when they occur. I hear a lot of hate, but in all, systemd seems to solve a ton of problems in a single place, which is something I think linux needed for a while.

jiveturkey2• 7 years ago

more like abomination

merb• 7 years ago

a lot of people shit on systemd because of the "SIGHUP"-"bug" however what they don't realize is that there are more use cases where logging out a user SHOULD kill all users processes.

I mean without that behavior linux is not a multi user system. The default is sane, however the lack of "permissions" in this kind is also a little bit bad, however since it is runtime configurable this should be enforced by the distribution, i.e. when installing the distri there should be a switch "this system is only used by a single user". it's basically the same as windows uac. while it looks bad for a single user, it's not that bad when you see computers as a tool which can have more than one user, with more than one "privilege".

system just shows that linux is by far not ready for the masses cause a lot of behavior is undefined, like the uid/username stuff. usernames can be created as you wish, but most programs will behave broken if you create usernames that start with a number, it's a security nightmare.

and the problem that there is no real process to actually get ALL people together who can solve this, it will never be fixed. there is only the posix standard but not everything applies to linux and not everything is a good idea in the sense of linux.

systemd actually makes a lot of things in the right way, but of course it's not perfect and probably at a certain point in time people would come up with something better.

Systemd as tragedy

Related Stories