Linus: “I no longer feel like I can trust ‘init’ to do the sane thing”

https://lkml.org/lkml/2017/7/6/577

by cnst • 9 years ago

350 339 9 years ago

Rjevski2• 9 years ago

I would assume this is about systemd. Personally my experience with it has been great - the occasional screw-ups (they did happen - something related to dbus made all systemctl commands fail until a reboot) were nothing compared to the time I gained not having to worry about prehistoric initscripts.

So while it isn't perfect, it's IMO a step in the right direction, if only by lowering the barrier to entry to actually using and managing the thing.

If you are worried about stability then you shouldn't rely on a single machine anyway. Personally I manage systemd screw-ups (though I haven't had one in production yet - all of them were on my personal machines while monkeying around with Archlinux) just like any other hardware failure - by making sure my app/service stays available even if I yank the power cord from the machine.

SwellJoe10• 9 years ago

It seems obvious he is referring to systemd. I like systemd, but Linus hates it when people break userspace, and systemd has broken userspace a lot.

Poettering seems to take the advice of "release early and often" to heart and pushes things into widespread usage long before they're quite fully baked. PulseAudio was awful for years, and then it was good and we all stopped complaining about it. systemd was kinda crap for months after it became widespread...a ton of breakage, including in ways that were occasionally dangerous. We're lucky, I guess, that systemd didn't take as long as PulseAudio to stop sucking all the time (systemd still sucks some of the time, but what it replaced sucked a lot, too).

So, yeah, you can't break userspace if you want Linus on your team, and systemd has broken userspace now and then.

asveikau2• 9 years ago

You seem to exist in a fictional universe where pulseaudio "got good" and doesn't suck. All these years later I have never seen that be the case. I think a much more likely explanation is that desktop Linux is less common since the rise of OS X in the "Unix-like developer machine" market so there are now fewer people noticing it sucks.

throwaway2048• 9 years ago

Its nothing to do with "release early and often", its gems like this that they refuse to fix over ridiculous hubris.

https://github.com/systemd/systemd/issues/6237

Its going to end up exactly like glibc under Drepper, where distributions are forced to have a mile of patches because obviously broken shit isn't patched because the maintainers are too arrogant to admit fault.

otterley4• 9 years ago

How, exactly, has systemd "broken user space"? That's clear neither from the lkml thread, nor here. I'm not sure what this discussion even has to do with systemd; and I'm not even sure how you can "break user space" from a user space program, which init is.

You might be able to modify your descendants' environment, but the rules under which that can happen are maintained and enforced by the kernel. It seems to me that if user space programs can do undesirable things, that's the fault of the kernel itself. To be sure, setting those boundaries and rules is one of the kernel's main jobs.

Qantourisc2• 9 years ago

I still hear a friend complain he often has to restart pulse because sound just stops working for him. And IIRC there is still no way to set the maximum amount of audio-buffer (for the end-user, not from the applications; some people prefer the occasional buffer under-run over the higher delay).

simias• 9 years ago

Well, pulseaudio is still pretty crap as far as I'm concerned. After an update last week Spotify won't work anymore and the sound in Firefox is glitchy. I haven't taken the time to investigate what caused that exactly, and I'm not exactly looking forward to it.

Not that I was a huge fan of ALSA either. It always felt a bit over-engineered for my taste. I can understand needing something like that if you want to make professional audio equipment with the Linux kernel but for my desktop use case OSS has always worked just fine for me and it didn't randomly break after an update.

As for systemd, it's bearable and it mostly does what it's supposed to do, but it still feels a bit unnecessary and its security track record is not stellar so far. I prefer the simplicity of the BSD init personally.

gcb02• 9 years ago

this is all nice when you ignore the tons of bug reports everyone else is getting because you decided to treat your users as testers.

why do you think linus is complaining? because he can't get sound to work on his own box or because kernel invalid bugs are over the roof thanks to bogus init?

gaius2• 9 years ago

There is a vast gulf between two communities, people who run Ubuntu Beta for fun on their own PC, and people who run 10,000 RHEL boxes in Prod, and both communities tend to talk "at" each other and misunderstand where the other is coming from.

The scenario you describe is something that would make one of the latter community recoil in horror.

derefr• 9 years ago

There's a third, rather small community: the distro-makers, who nearly to a one are—if not in love—at least hold systemd in high regard.

It's easy enough to understand their point of view: run through a Linux From Scratch install. You'll have to write your own init system. You'll be duct-taping a lot of things together, and your solution—while it might be something you're "proud of" in some sense—will definitely be lacking in engineering, no matter how much time you as one individual could ever throw at it.

After the LFS experience—which is essentially the distro creation experience—coming back to regular (pre-systemd) distros made one feel rather different: it was very easy to see how each distro was, at its core, just someone else's duct-taped-together init system with a pile of hacks accumulated on top. There wasn't any real Engineering; there was no shared knowledge-base of best practices for doing this or that. Each distro was a evolutionary Galapagos island.

From the LFS perspective, an init system that exists as its own standalone FOSS project, containing contributions from hundreds of interested parties and battle-tested engineering choices after coming-up-on-a-decade of use in virtually every scenario, is a wonderful thing. systemd is, finally, a reference implementation of an init system "at scale." If you want to build your own init system, you don't have to look at the init systems of 30 different distros to figure out what things each distro just did its own way for no reason, any more (which used to be "most things.") You can just start by looking at "what systemd does", and figure things out from there. Or you can use some systemd components, and some of your own.

petre6• 9 years ago

RHEL users can countinue to run systemd in production without disrupting the rest of the comunity with an init that goes beyond its scope.

pinpeliponni• 9 years ago

Perhaps. But then again, the RHEL people really need many of the features SystemD offers. The Ubuntu Beta people can keep running init if they must.

coldtea• 9 years ago

Because the latter community enjoys a hodgepodge of scripts doing arbitrary things instead of a standardized well organized mechanism?

mfukar2• 9 years ago

systemd has a few nice ideas, and it might be alright for whatever your use case is. I'm working on a product which uses it as its init system, and it's an endless source of problems. Its flaws are deep and unfortunately pervasive.

xaldir• 9 years ago

As a user, I would agree, systemd kinda just works and it's nice.

As a sysadmin however, I hate it so much. Not because I have to learn a new system which is nice, but because it breaks so much things. I had to rewrite scripts that just worked for years, I had to do a bit of black magic to make old software behave on newer distribs and it's just so frustrating. I get it that most of these things were not done in the right way but FFS, don't break my production stuff.

kochthesecond• 9 years ago

Oh yes, the init to systemd phase was brutal on arch. So many broken systems :(

knorker• 9 years ago

My problem with systemd is that is seems to have a policy (most certainly a history of actions, and that's really the significant thing) of NOT doing the sane thing, then being called out for it, and then having the systemd people defend not doing the right thing.

I can trust someone who makes mistakes. I can't trust someone who refuses to accept that they regularly make mistakes. Nor can I trust their work.

Aardwolf• 9 years ago

They're not really following the philosophy I expect to see a unix-like OS, so my experience also wasn't great. Like turning logs into a binary format, and then later bugging out on it so when you try to read it with their convoluted command (where cat or less was sufficient in the past) it gave errors on its own mess and refused to display anything.

Bugs happen, that's understandable, but simply the fact that they turn logs into a binary format shows their philosophy is different than the one I expect from a unix-like OS.

And almost all distros are following them sadly...

shimon_e• 9 years ago

Any reason why we can't have journaldfs?

raverbashing• 9 years ago

If you are worried about stability then you shouldn't rely on a single machine anyway.

Sorry, this is a complete BS "solution"

You don't keep a broken piece of software running then put more redundancy to make it kinda stable.

Redundancies are for when things break, not for when they're broken from the start

Nobody takes off from a plane with one inoperative engine, both have to be turning, so in case one fails the other one can take over

digi_owl13• 9 years ago

Effectively what you are responding to is the ever more prevailing "cloud" mentality, where there is always another instance to spin up in a second if something where to break.

http://www.commitstrip.com/en/2015/07/08/true-story-fixing-a...

And while systemd may have started on the desktop (likely to "solve" the speed of repeated bootups when using a laptop "securely") increasingly its development is dictated by this cloud mentality (as is the development of various distros, as someone basically shouted to my virtual face over at LWN).

btschaegg• 9 years ago

I could imagine that Linus is also influenced by experiences such as [1][2]. After a couple of issues like that I can imagine that he has his doubts about some of systemd's behavior.

[1] https://bugs.freedesktop.org/show_bug.cgi?id=76935

[2] https://lkml.org/lkml/2014/4/2/420

testcross• 9 years ago

The internals of systemd are maybe not the best. But the UI is definitely easier to grasp than what existed previously.

ill0gicity• 9 years ago

The UI is easier? I'm gonna need you to explain this.

Rusky• 9 years ago

Declarative unit files are far easier to manage than fragile ad-hoc scripts.

semi-extrinsic• 9 years ago

Come back and say that when you've tried using systemctl to find out why a service is failing. 200-line zero-content error messages are what finally made me purge systemd from my machines.

mdekkers8• 9 years ago

But the UI is definitely easier to grasp than what existed previously

Really? so you really think:

   systemctl <command> <service>

is an improvement over:

   [optionally, some initsystem] <service> <command>

Given that, personally in most cases, when I am actually messing around on this level, I will be repeating the above several times where the only thing changing will be the <command> part (so, it is handy to have this right under the backspace key)

The systemd UI is a hot steaming pile of garbage that has made my day-to-day life on the CLI harder, not easier.

jamespo4• 9 years ago

But I prefer running

    systemctl status <service>

and actually getting consistent output rather than whatever the init script developer chose to provide (if anything)

JdeBP16• 9 years ago

I suspect that you are not aware of what actually did exist previously. Mewburn rc had short pretty much declarative service definition files. Upstart had declarative configuration too, as did InitNG. Launchd, Upstart, and Solaris SMF had the launchctl/initctl/svcadm concept. daemontools/runit/perp/s6 had universal logging of standard output/error. And so forth.

* http://blog.darknedgy.net/technology/2015/09/05/0/

* http://jdebp.eu./FGA/run-scripts-and-service-units-side-by-s...

dijit4• 9 years ago

SMF without XML is gods init system.

It's so cleanly engineered, just like ZFS.. makes me sad that solaris died a death.

(Their package management was crap though tbh)

noway421• 9 years ago

I still haven't found an easy to remember command which prints journal log for the past 10 minutes.

While dmesg did that by default.

ausjke• 9 years ago

Like what happened to git in the past, will Linus write a new init himself this time?

saurik• 9 years ago

Linus wrote git because BitKeeper was taken away from him due to the Andrew Tridgell incident, not because he disliked BitKeeper; he didn't even mind that it was closed source.

speeder• 9 years ago

Decided to google about this...

Found this article: https://www.pcworld.idg.com.au/article/129776/after_controve...

I wonder, how many people reading that article would predict the rise of stuff like GitHub and GitLab and the success of Git to the point of crushing CVS and making SVN decline in popularity?

voidlogic• 9 years ago

I sure hope so... I've had to revert several Ubuntu machines to upstart and I have to "sudo service restart network-manager" 3 of 10 times my primary desktop starts (so I scripted it..)

Based solely on my personal experience (which is mainly Ubuntu desktop and server) Systemd is a hot broken mess. To me it feels like good goals with poor implementation.

I actually miss how simple init used to be... I understand the motivations for improving it, but if I had to choose ease of grokability or better boot times, I'll take grok every time.

glogla• 9 years ago

Given how many things systemd does and how this set of things is increasing with every version (lately including dhcp and ntp client, for example), it feels to me that the goal is to bring as much things under systemd (and thus Red Hat) as possible.

It's a sad day that the biggest Linux company is doing Embrace, Extend, Extinguish, but I guess I shouldn't have expected more from RH.

RJIb8RBYxzAMX9u2• 9 years ago

Please no. That would be like if Gandalf wielded the One Ring. I've no doubt the result would be very functional and well engineered, but it would be awful in other ways.

kobeya• 9 years ago

That would be the best possible outcome.

pwdisswordfish2• 9 years ago

Unlikely, for the same reason that he didn't tackle application packaging[0]: because writing a successful init system would necessitate either perfect backwards compatibility with shell scripts (next to impossible unless you're basically just copy-pasting sysvinit) or a massive coordination problem of rewriting the long tail of all the crazy different daemons people run to be compatible with the new system.

Git didn't have this problem because:

0. Version control systems don't have to interact with as many other software packages anyway (editor/shell integration is nice, but you can live without them and just run git directly from the command line);

1. There were compatibility layers that let Git talk to popular VCSes;

2. The fact that you use Git personally does not affect anyone else, obviating the bulk of the need for coordinated upgrades (there were stories, from before GitHub was a thing, of 'guerrilla Git users' in companies that officially used CVS).

These three things ensured that a relatively painless incremental upgrade path existed; in the case of init systems, it will be much harder to establish.

[0] He talks about it in https://www.youtube.com/watch?v=1Mg5_gxNXTo ; can't find the timestamp right now, but it's in the latter half of the video IIRC.

JdeBP16• 9 years ago

Considering the broad range of such softwares that already exist, written by people such as Felix von Leitner, Gerrit Pape, Laurent Bercot, and so on, why do you think that Linus Torvalds is the person to write one? Or needs to write one?

* http://blog.darknedgy.net/technology/2015/09/05/0/

* http://skarnet.org/software/s6-linux-init/

* http://smarden.org/runit/runit-init.8.html

* http://www.fefe.de/minit/

* https://jdebp.eu/Softwares/nosh/guide/system-manager.html

noisy_boy• 9 years ago

There were many version controls systems before Git came along. He doesn't need to write it. But if he wants, he has the technical chops to do a damn good job of it.

TheAceOfHearts3• 9 years ago

Here's something I don't understand... Why do so many people hate systemd but not macOS's launchd [0]? Don't they serve similar purposes? I've been digging into macOS for a few months now, and I'm pretty happy with their defaults.

EDIT: I don't know why I'm being downvoted for asking a genuine question. I'm not a linux expert, so I'd love to hear your criticisms.

[0] http://www.launchd.info

otterley4• 9 years ago

I suspect it's because nobody ever really has to touch launchd configurations because Macs aren't commonly used as Internet servers.

TheAceOfHearts3• 9 years ago

I've been happily using macOS server at home for a few months. It's not a production-ready deployment, but I haven't had any complaints.

Aside from that, using it on my personal laptops has worked surprisingly well. Once I familiarized myself with the CLI tools, I haven't had any issues.

dredmorbius9• 9 years ago

Anecdata: I wanted to set up a Mac box as a remote syslog server. It's running, IIRC, rsyslogd, which is to say, remote syslogd.

All you've got to do is tweak the PLIST.

Read the manages. They're out of date.

Looked at the PLIST tools (not my thing). Doesn't seem to be an option.

There used to be, back when PLISTs were actual files you could edit. That's gone away now though, best I can tell.

Got a Linux box up and running, that now accepts remote logs.

stephenr9• 9 years ago

The use of non-executing service definition files isn't really the problem with systemd - it's one of the parts I actually like.

The problem is more about the project itself. The scope keeps creeping to consume more and more things. The developers have a "we know better, do it this way, your argument is irrelevant" type attitude to outsiders.

Qantourisc2• 9 years ago

Indeed your last argument is my beef with systemd as well. I describe it as:

"Systemd is great! But it's an inconsiderate piece of software!"

Meaning it work nice, it does it's job it does it fast. But if you want to things your way, or want to run something not-systemd standard, you are in for some fun.

watwut2• 9 years ago

My understanding is that interpersonal drama and similar BS has large role in it. Note how often is the anger targetted at main dude behavior. Also some don't like change, react aggressively to it and systemd is change.

There are also technical reasons, but those tend to attract less of pure anger.

notzorbo3• 9 years ago

how often is the anger targetted at main dude behavior

That anger comes from somewhere. We don't just go around picking random people to become angry with. Poettering has a lot of influence in core components of Linux (dbus, pulseaudio, systemd) and he's a bad developer. His designs are horrible, his implementations buggy and shoddy and don't fit into the overall architecture of the rest of the system. So, yeah, there's anger towards him and I think it's more than deserved.

Also some don't like change, react aggressively

We like change just fine if it brings improvements. If not, we don't like those changes. Again, we don't go around hating on random changes.

There are also technical reasons, but those tend to attract less of pure anger.

All of the anger is caused by technical reasons. Whether it's shoddy programming or shoddy reasoning about technical issues on the systemd developers part. Almost no-one personally knows Poettering or the systemd developers, yet we still think their software sucks.

AceJohnny24• 9 years ago

I've worked with launchd, and I hate it more than systemd. Most of it is because of launchd's atrocious doc. Note that launchd was rewritten a few versions ago for Reasons [1]. I couldn't find any release notes that highlighted this, and you wouldn't know it from launchd's version number...

Thing is, launchd's developers made a good effort at preserving backwards compatibility, but it's far from perfect. And most online docs (such as launchd.info that you're linking to) is subtly out-of-date. For example, I see no mention of "service-targets": new launchd learned about "bootstrap domains" (huh?), which are namespaces implemented at the Mach (macOS kernel) level. It's not a Unixy thing. Trick question: how do you define a user job that runs in the background even when the user's not logged in? How do you define a user job that runs if the user logged in via SSH (ie not GUI)?

It's late and I'm tired, I can try being more coherent tomorrow. There are some good sources of information hidden out there, but the first few Google responses are woefully obsolete. And fuck Apple's own docs.

[1] I'm not sure what the reasons are, but I suspect for a) merging iOS and macOS functionality and b) tightly integrating it with the kernel's XPC interprocess comms infrastructure (again, for iOS?)

mmmpop• 9 years ago

> I've worked with launchd, and I hate it more than systemd. Most of it is because of launchd's atrocious doc.

I had to do some provisioning work with NFS and pf and the docs were maddening. I never actually got VirtualBox and NFS to play nicely but once I got a couple little quirks of pf it was pretty reliable.

eridius5• 9 years ago

do you define a user job that runs in the background even when the user's not logged in?

Well, you can't. Only administrators get to define things that run when the user isn't logged in. That's an intentional decision.

How do you define a user job that runs if the user logged in via SSH

Offhand, I don't think you can do that either, though I haven't researched this particular angle. And it would be surprising to me that SSH'ing into a machine would trigger daemons anyway.

dredmorbius9• 9 years ago

I'm glad to see I'm not the only one to have this experience.

I've not mucked much with OSX admin, but the times I have, it's been fairly ugly. PLISTs particularly so.

merijnv• 9 years ago

Most of it is because of launchd's atrocious doc.

Really? I read up on launchd a year or so ago to setup backups and I though the documentation was amazing...Which docs are you referring to?

nandemo• 9 years ago

Why do so many people hate systemd but not macOS's launchd [0]?

My guess is, it's not as relevant because Mac OS isn't as widely used in servers as Linux.

digi_owl13• 9 years ago

Also, MacOS/OSX is one OS with one developer.

Systemd is effectively Freedesktop/Fedora/Gnome dictating how every last Linux distro is supposed to behave. And doing so by invasive dependency chains.

Observe how LFS now have two parallel books. One continuing the one they have maintained from the start, and another dedicated to systemd. This is a good indicator how invasive systemd is in the basic construct of a Linux distro.

ori_b• 9 years ago

Don't they serve similar purposes?

In theory, yes. In practice, systemd reinvents half of the rest of the system, poorly. Even to the point of shipping a half assed dns resolver and a dhcp client that doesn't even support lease renewal, and when people file bugs the invariable response is to blame the user.

nullc• 9 years ago

ntp client too, which at least one point had hardcoded defaults in the distribution source that gave incorrect time.

jandrese3• 9 years ago

I don't even understand how someone could ship that DNS resolver, especially as the default install. It's completely broken on my systems, literally crashing on every other lookup. Nobody online seems to have a fix either, I just had to disable the thing and revert back to the old resolver.

vbezhenar• 9 years ago

I don't like both. While I can understand the need for complex configuration on macOS: it runs on laptops, my Linux runs strictly on server, reboots once in few weeks or months and I don't care about boot time (my BIOS boots for 5 minutes, I could care less if Linux will boot for 10 or 60 seconds) at all. Good old init with bash scripts worked wonderfully, was transparent, easy to understand, hack and extend. While I can live with systemd, it's a step in wrong direction.

dredmorbius9• 9 years ago

Having worked with inits of BSD, sysv, upstart, systemd, and launchd varieties, I strongly prefer the administrative and debug capabilities of the script-oriented ones.

BSD's choice to drop everything into a single rc file is not particularly elegant, but it can be coerced to do what you want. The sysvinit user interface, frankly, isn't bad:

    /etc/init.d/<service> [start|stop|reload|restart]

Need to debug?

    /bin/sh -x /etc/init.d/<service> [start|stop|reload|restart]

I've been known to copy the files elsewhere to work with them, if necessary. The last time that was necessary was well over a decade ago. As a sysadmin (1 - 15k boxen), broken init scripts simply weren't on my radar. (Neither was boot time, or network management, or device management, or logfile management, or ....)

I'll add that my distro of choice is Debian (or derivatives, which I generally mean here). One thing that I noticed when first switching to Debian, back in the mid 1990s, was that its init scripts were vastly easier to follow than Red Hat's. Debian has a fairly standard structure (/etc/init.d/skeleton is the bare-bones template), and most scripts follow that within some reason. There are a few exceptions -- scripts which initialise things (bootclean, mount, networking) rather than start services, tend not to look like service initialisation scripts, on the inside. On the outside, though, they do.

Further: once configured properly, those scripts don't change. Debian in particular does an extraordinarily good job of extracting out all necessary configuration to an /etc/ configfile (which it then won't touch unless you specifically tell it to).

My experience on MacOS (see elsewhere in this thread) is that stuff which ought be easy isn't. My experience with systemd, to date, hasn't evidenced this, as I am largely using the extant sysvinit scripts. But I've noticed numerous things Wot Used To Work ... do not. And the response from the sysvinit devs, particularly from the top, as well as various sycophants on numerous fora, has been exceedingly disturbing.

Case in point, my single most controversial Reddit comment:

https://www.reddit.com/r/linux/comments/2dvmdn/what_do_you_a...

(No comment on the general tendency of crowdsourced moderation systems to vote for popularity rather than truth.)

JdeBP16• 9 years ago

BSD's choice to drop everything into a single rc file

... is a description that is out of date by coming up to 2 decades. Mewburn rc came along at the start of the 21st century.

* http://blog.darknedgy.net/technology/2015/09/05/0/

lobster_johnson• 9 years ago

Maybe you're thinking of /etc/rc.local, which is a script on *BSD where you could just dump random commands, which would then be executed at startup. Last I checked, this mechanism is still supported, but not really the way to do it.

ktRolster3• 9 years ago

Every init system ever invented has been criticized. That's why there are so many of them.

gruturo• 9 years ago

I hate systemd because it's horribly, cancerously invasive, often messing with existing functionality that works well and replacing it with worse substitutes / duplicates.

In recent memory, off the top of my head, it messed with NTP, DNS resolution, Syslog, DHCP, Nohup/Tmux, not to speak of taking over udev so it won't run without it (thanks for eudev).

owaislone2• 9 years ago

Not nearly enough engineers care about MacOS at the same emotional level as Linux. People who use Mac usually don't care much about internal architecture but their actual experience while Linux users who are also developers are extremely vocal of every single component inside Linux

vacri2• 9 years ago

Or it could be that MacOS doesn't run on servers, where Things Just Have To Work. MacOS has the luxury of always having a ready pair of hands right there to fiddle with things. The people who care about systemd one way or the other are the people who care for servers.

Look upthread at the 'nohup and logout' "bug". That's not something you're going to encounter on MacOS, because it's not something you do on MacOS.

dijit4• 9 years ago

Not sure why you think people don't dislike launchd.

The problem with systemd was that there was potential to be better; the idea is good but the core developers reek of hubris.

Launchd is OSX, and osx has some horrors but that's ok because they're desktop systems and you don't work on fleets of thousands of them usually.

There was a time where Apple were threatening to upstream it to FreeBSD- but aside from that I think the consensus is that launchd is quite horrible too. It's just systemd could have been so much better and not tried to gobble up the entire Linux ecosystem.

dsfyu404ed2• 9 years ago

Because the people who know enough to critique a init system generally run linux on all their devices.

tw04• 9 years ago

For the same reason an airline mechanic doesn't care about the location of the fuel injectors in his car, but cares very much about the location on the engines he works on all day.

I won't claim to speak for everyone here, but there's almost nothing I install third-party that I care about during the startup process (on my laptop). And even if I did, they're generally closed-source apps I can't access the source of anyway. 99% of my day-to-day usage of a laptop is with applications I'm manually starting and stopping.

I care VERY much about services automatically starting properly on a server. 99% of my day-to-day usage of a server is applications that automatically start and stop.

jandrese3• 9 years ago

For me the problem with SystemD is that it seems to be trying to make Linux run like Windows.

If you are running in a generic "This is my machine and I just did the normal install for a regular use case then everything works" it seems great. But the instant you try to do something a little out of the ordinary it becomes a absolute nightmare of opaque message passing interfaces and hours on Google.

A few examples: 1. In the old days you could network boot a machine with a kernel compiled with a couple of flags turned on and an NFS share. SystemD systems can be network booted, but you're going to spend a lot of time poring over obscure forums finding all of the tweaks you need to make, and it will be a lot slower and toss up error messages for things like UUIDs changing on different systems.

2. If you want to have the wireless card change MAC addresses before it connects for slightly improved anonymity at public hotspots, you had better be willing to upgrade to the bleeding edge. NetworkManager broke the existing solutions for doing this and implemented it's own model, but it required a version of wpa_supplicant that was never really released so you had to build it yourself from a snapshot and install it and tell your distro not to update it. The SystemD devs got tired of waiting for the wpa_supplicant devs to release the new version so they moved the functionality into their code. At the same time they pulled in the DNS resolver but broke it so it crashes on literally every other name lookup.

3. NetworkManager pulled in OpenVPN support as an option. It's awesome when it works, but when it dies it refuses to give you any sort of clue as to why the connection didn't work. You just get a generic "it failed" message like in Windows. Not "Authentication Failed", "Cannot connect to Server", or "Clocks unsynchronized", not even a cryptic error code. Every single error results in just a generic message and you have to either hope the logs have something useful or more often you have to fire up Wireshark and inspect the authentication exchange by hand to figure out what is up.

Lord help you if you find yourself with a machine that hangs at boot because some process didn't receive some message it was expecting and now you have to figure out who was supposed to send that message and why they didn't send it. This isn't the old days where you could trace shell scripts to figure out where the problem was, now you have to know the whole thing before you can fix anything.

And don't get me started about moving the syslog into some binary database thing that you can't grep without going through some export process.

gjvc• 9 years ago

The quote here in the title is missing the vital context -- the full quote is

""And yes, a large part of this may be that I no longer feel like I can trust "init" to do the sane thing. You all presumably know why.""

throw2016• 9 years ago

Making too many assumptions about the user environment is not smart. Then you start putting constraints and becoming defensive when these assumptions are questioned. This is becoming a pattern for systemd as is blaming everyone else. [1]

Those who need the complexity or some features should adopt the technical debt, there is zero rationale to enable it by default and impose it on everyone.

For instance if you have too many network interfaces and naming is a problem then switch on predictable network names which btw is anything but predictable for human beings and let the 97% who don't carry on.

12 digit random names are simply not predictable or better for human beings than eth0, wlan0? If there is a problem predictable network names is not the solution.

Similarly if you need binary logging and an audit trail then turn on binary logging but don't impose it on everyone else.

Its time to move beyond silly accusations of 'hate' etc every time there is a discussion about systemd because at the moment it just serves to deflect criticism and avoid accountability for questionable decisions.

[1] https://github.com/systemd/systemd/issues/2407

microtonal• 9 years ago

Those who need the complexity or some features should adopt the technical debt, there is zero rationale to enable it by default and impose it on everyone.

Indeed. Even if I have multiple ethernet interfaces, I'd prefer to map a MAC address to eth[0-9] myself. But most machines only have one ethernet interface and one WiFi interface anyway.

These things pop up all over the place. E.g.: why would I want to have NetworkManager on a server, which never switch networks and need to set up a VPN connection? Why do I need firewalld on a server? I rarely need a zone-based firewall on my laptop (probably like most users, I just block all incoming traffic), let alone a server.

Layer upon layer is piled to solve hypothetical situations that do not arise for > 90% of the population.

(I do like systemd as a service manager, but systemd and the surrounding ecosystem has made it terribly difficult to understand and debug UNIX systems.)

mdekkers8• 9 years ago

why would I want [insert a systemd design decision]

Because systemd wants to be The One Ring of Linux, and tries to be all things to all systems and all people.

marcosdumay2• 9 years ago

Most of my machines have more than 1 wired interface, and guess what? Those predictable names have caused me way more problems than they ever fixed. And those problems have always been harder to fix. Just stable names, like udev used to do were enough for nearly everything.

why would I want to have NetworkManager on a server, which never switch networks and need to set up a VPN connection?

It's funnier because NetworkManager must wait for the filesystems to be mounted, but then, networked filesystems (that are, those ones, important there) need a functioning network that NetworkManager postpones to after it's up.

grandinj2• 9 years ago

Poettering has a strong bias towards making sure bugs get fixed in the right layer, instead of creating forever-living-workarounds in higher layers.

In the bug you reported, however, I noticed that he pushed a commit to work around the underlying bug, so he's not completely impractical.

digi_owl13• 9 years ago

Unless he can introduce a layer he controls. Observe that rather than work with Alsa to find some way to handle transient audio devices, he introduces Pulseaudio.

boomlinde2• 9 years ago

Another example of this type of defensive response to bug reports:

https://github.com/systemd/systemd/issues/6237

systemd runs services as uid 0 because it can't parse a valid user name (of a non-uid 0 user). Of course it's not systemd's fault. It's not even a problem at all.

stephenr9• 9 years ago

I fucking hope not.

Linus is clearly very smart, but user-facing tools is not what he's good at. Git is a perfect example of this.

Mercurial and Git were created ridiculously close to each other, and their main features/use model are quite similar, and yet mercurial is literally years ahead in terms of usability. The results to a given command are 99% of the time, what you would intuitively expect.

Now here's what I would love to see: Linus initiate a fork/rewrite/whatever of the init part of systemd, that keeps the concept of non-executing service definition files. They don't have to be compatible with systemd but that wouldn't necessarily be a bad thing either.

Edit: actually, aiming for compatibility probably means having to reimplement systemd weirdness, so probably is a bad thing as I think about it.

beefsack• 9 years ago

It's quite frustrating seeing people complain about Git usability like it's some broadly accepted and objectively bad thing. As frivolous as anecdotes are, I took to Git very smoothly and I found it to operate in a very obvious and explicit fashion, particularly once I'd shifted my mental model from what I was using at the time (primarily SVN.)

Git is a very powerful, flexible, reliable and efficient tool. I'd be over the moon if someone wrote an init system holding similar values.

stephenr9• 9 years ago

That's the thing: it is broadly accepted as having poor usability compared to other version control tools.

If you haven't had those issues, you're one of the lucky ones.

Edit:

Git is a very powerful, flexible, reliable and efficient tool.

I didn't say it isn't any of those things. I said it's harder to use than competitors such as mercurial.

AndrewBissell• 9 years ago

Couldn't agree more here. I picked up git at the same time I was starting out as a professional programmer, and never had any major problems with it at all. In fact I remember it being a breath of fresh air from my previous version control experience, which was banging around with TortoiseSVN on Windows.

I still remember the first time I used "git cherry-pick," I just guessed what it needed from my (then pretty limited) git experience, and it just worked seamlessly. The few little command-line quirks in git (like "checkout" being used to create new branches) are barely worth mentioning.

pwdisswordfish2• 9 years ago

The UI has some warts, and a few design choices were made that some people find questionable. You can get used to them (which is probably why most people don't even notice it as a problem), but they're there.

The best survey of Git's quirks that I've seen is, unfortunately, http://stevelosh.com/blog/2013/04/git-koans/

cthalupa3• 9 years ago

As a not-developer that has to put on a developer hat and learned git and mercurial at about the same time without any prior experience working with cvs/svn/etc outside of checking stuff out from sourceforge, I don't find git to be any more difficult to use than mercurial

laurent123456• 9 years ago

As far as I understand Linus was mostly involved in creating the git backend, and this part is actually well designed. It looks like something went wrong during the UI development, maybe they've tried to follow the underlying backend too closely instead of thinking it in terms of how users manage their source code.

dang2• 9 years ago

We detached this subthread from https://news.ycombinator.com/item?id=14733364 and marked it off-topic.

stephenr9• 9 years ago

I don't understand how it's off topic? It was a reply to someone asking if Linus would write his own init (because he's unhappy in systemd), and my take that it would be a bad idea because of the usability issues in the last big thing he started.

cperciva• 9 years ago

I'd say that I no longer feel like I can trust 'git' to do the same thing either... but who am I kidding? I never trusted it to do the sane thing.

dijit4• 9 years ago

You can say some negative things about git, but sanity is not one of them.

zdw2• 9 years ago

systemd is a mess. A while ago I tried writing a script for CD ripping that ran when udev detected a music CD in the optical drive, ripped the CD, then ejected the disk.

This kept failing. It turned out to be systemd having a hardcoded timeout of something like 30 seconds on any process launched by udev, that couldn't be changed without recompiling it from source, or having it fork off some other process group that lacked this limitation.

From what I can tell, systemd has some of the worst sort of developer myopia, where whole swaths of legitimate use cases are actively destroyed.

rlpb• 9 years ago

It sounds like you're blocking udev while your script executes, and instead you should start a service from your udev script and then exit. For example, see systemd-run(1).

...where whole swaths of legitimate use cases are actively destroyed.

Your use case is fine. Your implementation doesn't fit with udev's design; that's all.

jononor2• 9 years ago

And I am fairly sure that this design predates systemd by several years.

zdw2• 9 years ago

I was having udev starting a separate ripping script as a background process, and it worked on pre-systemd udev systems.

systemd kills the entire udev process tree (all children) including the ripping script after 30 seconds.

yuhong• 9 years ago

I wonder why Poettering chose Kay Sievers to work on systemd in the first place.

theonemind2• 9 years ago

Linus has the proverbial moral authority to bootstrap an init system and the technical skill to pull it off. Amusingly, this strikes me something like the chairman of the Federal reserve hinting that he thinks interest rates are too high. Hopefully, the systemd developers will take notice and clean up some of their act. (Not to insinuate anything. They have widely known problems in playing nice with other projects, taking feedback, etc.)

SolarNet• 9 years ago

waits patiently for Linus to get fed up with ___ enough to write it himself

dies

theonemind2• 9 years ago

I don't really think he would, will, or wants to. But that he could, and actually might if he begins to feel that systemd hurts Linux, might create some positive pressure on the systemd developers. That's exactly the thing, like the fed. They have enough power that an implied threat can fix things (sometimes to some extent) without them actually taking any action.

striking• 9 years ago

I mean, he wrote git for that reason.

dredmorbius9• 9 years ago

At this point, less likely on systemd devs than those who employ them (Red Hat) and distro maintainers (Debian, Ubuntu, Fedora, Gentoo, Arch, etc.).

throwacct7835• 9 years ago

Actually Gentoo still uses sysvinit by default, which is one of the reasons I like it. They also maintain a fork of udev (eudev) that doesn't require systemd.

dredmorbius9• 9 years ago

Ted Ts'o, the Linux ext2/3/4 filesystem developer, posted this two days ago at G+ as well. Interesting discussion, particularly about 1) systemd's bad programming taste and complexity, and 2) Lennart Poettering's inability to admit when he is wrong.

"For me a lot of "good taste" is about adding complexity when it's needed, and avoiding it when it's not needed. And if you do have complexity, making sure you have the tools so you can debug things when they break. And for me one of the things that I don't like about systemd is that it has added a lot of complexity, and when it breaks, trying to debug it can be almost impossible."

Also:

"Heck, I don't even I want to file bug reports, just so I can get abusive messages from Lennart. At least when Linus flames me, it's because I did something wrong which hurts users and which I d*mned well should have known better, given my years of experience in the community. Lennart just flames you because you're wrong, and he's right. By definition."

And:

"The high bit, in my opinion, is "not being able to admit you are wrong". If someone (such as Lennart) is always right, then it's impossible to have a technical discussion in a post-mortem to avoid similar problems in the future. In a company, if there are personnel issues like that, you can escalate to the person's manager, or use other mechanisms. In the open source world, all you can do is route around the damage. Whether you call the inability for someone to admit that he or she has contributed to the problem a "lie" or just that they were "mistaken" is really beside the point as far as I'm concerned."

https://plus.google.com/+TheodoreTso/posts/EJrEuxjR65J

digi_owl13• 9 years ago

Ts'o strikes me as damn close to Torvalds in attitudes. The guy have been with the kernel for almost as long as Torvalds himself, happily developing EXT version after EXT version for it.

The problem is with more recent people, that seem to be more "american" their approach. Always looking to climb the ranks and find new domains of development to undertake rather than stick with one area and become deeply proficient.

eleitl• 9 years ago

Gee, such a surprise. Why didn't you speak out about it earlier, then?

bigato• 9 years ago

I was thinking exactly this as I was reading the message. Linus was the one voice that could probably have prevented systemd from taking control. This comment strikes me as too little, too late. I just hope that the transition back to a saner solution, if it will ever happen, won't be too painful now. Or it could trigger mass migration out from Linux. I, for one, am living happily in BSD land by now.

digi_owl13• 9 years ago

Because he do not want to get involved with userspace bikesheding (beyond posting rants about insane defaults in his distro of choice).

He sees his domain beginning and ending with the kernel, and that's it.

Sadly certain people below him, that he trust deeply, are not so content (some even support Poettering and crew). And i fear the day he hand off the keys to the kernel to them.

reactor4• 9 years ago

Linus giving his opinion on userland is a far cry from bikeshedding though...

qb454• 9 years ago

Just wanted to say that the issue is about automatically copying rlimits from the init processes to all processes executing setuid binaries - apparently Linus thinks that init could possibly apply some rlimits to itself for its own purposes which wouldn't necessarily be suitable for copying to any other random process.

IMO this idea with copying is in itself even less sane than init using rlimits on PID 1.

teddyh• 9 years ago

This is taking a quote out of context and trying to start the old usual systemd flamewar. Flagged.

dpark5• 9 years ago

This isn't out of context. It's a direct quote and links to the full content of the email.

Also "If you flag something, please don't also comment that you did."

https://news.ycombinator.com/newsguidelines.html

owaislone2• 9 years ago

They [systemd] shouldn't piss him off too much. He will sit down for a week, come out with his own init system which will take over the world. /s but not entirely :)

dsfyu404ed2• 9 years ago

This reminds me of a relevant joke.

Systemd walks into a bar. It shoots the bar owner and proclaims itself to be the new owner. It then turns the bar into a farm, brewery and distillery, opens a casino, a freight rail line and an investment bank. Oh, and there's an init system thrown in there somewhere too.

floatingatoll4• 9 years ago

Is there any useful comment anywhere on this post that talks about the technical meat of Linus' post here? He explicitly goes out of his way not to go into a long-form argument about systemd, and then HN dumps something like 150 points of comment karma into systemd.

EDIT: Here are links to comments discussing the actual content of his email, in case anyone else came here for that and left wanting.

https://news.ycombinator.com/item?id=14734868

https://news.ycombinator.com/item?id=14733973 -> https://plus.google.com/+TheodoreTso/posts/EJrEuxjR65J

EDIT: I read all the comments on this post, and I am disgusted to report that (as of this EDIT, nitpickers) the 2 above-linked comments are the sum total of actual replies to the content Linus was trying to discuss. I hope LKML had more self-control than HN.

EDIT: 14733558, it was really excellent of you to try and defuse things, too.

smhenderson• 9 years ago

Yes, he specifically goes out of his way to refer to init as generically as possible. But then drops this line at the end of the post:

And yes, a large part of this may be that I no longer feel like I can trust "init" to do the sane thing. You all presumably know why.

It's not hard to understand what he meant and for some of us it's articles like this on HN which give an opportunity to talk about systemd without being off topic.

IMO the take away from the article is that systemd has become enough of a moving part that it would be unwise to rely on it for default anything. Maybe I read it wrong but there's my take on the technical argument against using init to set default limits on processes. There are already defaults in the kernel so use them and allow an override for admins that need one.

floatingatoll4• 9 years ago

It's impossible to misinterpret that he's referencing systemd, but that's not the point of his post. He's saying, "Kernel over here, Init over there, One way transaction". You're talking about whether Systemd is qualified to be init. He's talking about whether the Kernel-side or Init-side is responsible for defining a given value. He declares his bias clearly (trusts Kernel, doesn't trust Init, especially doesn't trust Init via Systemd) and then makes his point. I'm just deeply saddened that literally two comments out of a hundred bothered to discuss his point, that divide, at all.

qb454• 9 years ago

for some of us it's articles like this on HN which give an opportunity to talk about systemd without being off topic.

I would argue it is off topic because it's barely relevant to the submitted link. Just post "Tell HN: systemd sucks" and you can talk systemd all day long.

You could even include some actually real and concrete concerns, like trying to bloat the kernel with some linux-on-teh-desktopz IPC which will probably be replaced by the next big thing by 2025, bundling some weird syslog replacement, bundling a half-assed DNS cache (it was originally vulnerable to spoofed answers right off the bat, besides the recent remote code execution oopsie), polluting dmesg with systemd log messages, asking the kernel to workaround systemd crashing when the kernel is booted with debug output, Poettering's dismissal of the UEFI bricking fiasco, the recent failure to drop privileges on usernames starting with 0, ...

You guys behave as if some overblown pretext were needed to flame systemd :p

dpark5• 9 years ago

I don't understand your complaint, especially given that your last two links are more discussion about problems with systemd instead of the divide you think is the interesting point.

floatingatoll4• 9 years ago

I apologize, I'm not sure how I could improve it to make it more clear.

jwilk2• 9 years ago

For those who can't stand lklm.org UI, here's the mail on spincs.net:

https://www.spinics.net/lists/kernel/msg2550717.html

Tharkun• 9 years ago

Systemd is a steaming turd. However, it's one we have to live with, and there doesn't seem to be anything better on the horizon.

A good book on the subject might make things slightly better. However, it doesn't seem like anyone is interested in writing one :-(

codesurfer• 9 years ago

When so many think that something is wrong, maybe there is some truth about it...

If even the creator have concerns, man, that's not a joke!

Linus: “I no longer feel like I can trust ‘init’ to do the sane thing”

Related Stories