Why I dislike systemd

http://www.steven-mcdonald.id.au/articles/systemd.shtml

by geoffbp • 11 years ago

174 244 11 years ago

steven-mcdonald.id.au

ChuckMcM2• 11 years ago

Steven, if you're reading, you got your history slightly wrong. System V got its init handling from Sun's SunOS 4.x. It was negotiated as part of the integration between SunOS and System V which would become the "one UNIX for everyone". Unfortunately for all of us, AT&T really really loved "run levels" and Sun really really liked BSD's way of configuring the system. Which got you the unholy love child you saw in Solaris 2.0 and System V release 2.0.

For the other Sun folks from the systems group who quit rather than submit to the demands, you were right, I was wrong. We couldn't convince them of the error of their ways.

yuhong• 11 years ago

This history looks wrong. Eg it was SVR4 that merged SunOS

ChuckMcM2• 11 years ago

You are correct it was System V, release 4 that merged with SunOS. It was also the release that Sun and AT&T "combined" the BSD init/rc system with the System V init/run-level system.

StevenJM• 11 years ago

Thanks for the info. I've updated that section of the article with a note (and a link to this thread).

tilt_error• 11 years ago

My eyeballs just exploded... Unreadable.

dredmorbius3• 11 years ago

Readability's bookmarklets help: https://www.readability.com/bookmarklets/

(But, yeah.)

skrause• 11 years ago

I recommend the "zap colors" bookmarklet from https://www.squarefree.com/bookmarklets/zap.html

It simply changes all colors so that text is black-on-white, but leaves everything else unchanged.

Reefersleep• 11 years ago

The newest version of Firefox has a little icon resembling an open book at the right side of address field that does something similar. Quite handy :)

Qantourisc• 11 years ago

Yellow on black is a very readable/contrast colour. However I can believe the yellow is offensive on some screens/peoples eyes.

http://www.steven-mcdonald.id.au/ugly.shtml :D He should still ditch the yellow though.

Sapient• 11 years ago

It was the first time I have used the Reader View on Firefox and I was really impressed with it.

unexistance• 11 years ago

I used same bg/fg colors everywhere in firefox, solve the issue long time ago :D

erikb2• 11 years ago

I have the opposite experience. Thanks to the dark background my eyes were way more comfortable than on HN for example.

empressplay3• 11 years ago

I don't "hate" systemd either, I just think it's a bad approach. It flies in the face of the "open architecture" concept and stifles innovation. It discourages users from learning about how their operating system works, and it unnecessarily puts too much control of a fundamental component of the OS in the hands of a too-small number of people.

None of these reflect the Linux I grew up with. Thus, I am not a fan of systemd. #notmylinux

cubancigar112• 11 years ago

ossreality you are [dead]

olau• 11 years ago

Interesting way to think about it.

IMHO systemd is taking much of the hacks and weird glue that has historically held distributions together and replaced it with an active upstream project. Maybe some of the solutions they've come up with could need some improvements, but I have much higher confidence those improvements would happen now compared to how things used to be.

Nowadays you can code something up and come to the systemd mailing list and get feedback and get it integrated. Previously you'd have to convince one or more not-really-connected distro hackers that your new fancy thing is worth rocking the boat for.

weland19• 11 years ago

Unfortunately, the systemd development team is not that easy to work with, which is part of why a lot of people distrust systemd the way they do.

In theory, yes, it should be the way you mention it. In practice, even bug reports end up unacknowledged and labeled as features after a few philosophical debates.

No one who has used more than one Linux distro would disagree that a unified system to control init services is a good idea. It's this particular implementation that's problematic, not the principle.

johnny22• 11 years ago

Definitely disagree with that. I've been using some form of a Linux based OS since 1998. Now i'm finally starting to feel like this is #mylinux.

exelius3• 11 years ago

Once upon a time, we were obsessed with efficiency and speed of code. We created powerful, but simple systems like sysvinit, and we designed to optimize speed in 80% of use cases. The Linux you grew up with used to mostly run on x86 machines, except the majority of Linux devices now run on ARM with different power requirements, hardware, drivers, etc. than x86 ever had to deal with. The increasing complexity in the use cases of Linux is driving systemd, not some desire to stifle innovation (which IMO is exactly the opposite of what systemd is doing).

Our CPUs got fast enough and our use cases so diverse and network dependent (leading to lots of idle CPU time) that we stopped worrying about raw performance above all else. In nearly every case where you're not doing something glaringly stupid, performance is a function of the complexity of your problem set. Things like power consumption and safe asynchronous communication are more important than raw speed, and these things need to be coordinated through the same interfaces on every Linux machine (though not necessarily through the same methods).

Systemd doesn't so much fly in the face of "open architecture" as it does introduce architectural concepts that are sufficiently abstract that much of the community doesn't fully understand them. But this is complexity for the sake of flexibility -- which is far easier to deal with than complexity for the sake of performance. Best practices around specific use cases will eventually become common knowledge.

vezzy-fnord3• 11 years ago

Your implication is that sysvinit was designed for speed. I can't seem to find much evidence for that.

In fact, sysvinit's purely synchronous nature and its reliance on the patterns of symlink farms to represent system states in the form of "runlevels", along with the tendency to write bloated shell scripts for it because no one ever bothered to write a proper rc/service runtime framework until much later (in contrast to the BSDs), meant it was slow.

IIRC, one of the motivations for some distros to symlink /bin/sh to dash was to reduce boot times. Then we got ugly hacks like startpar for asynchronous script executions, insserv for parsing dependency directives inside scripts, and Debian had all the cruft behind sysv-rc just to make managing the runlevels easier.

The whole thing descended into an unmaintainable mess that performance improvements in systemd and other alternative systems were unsurprising.

keypusher2• 11 years ago

I agree with most of this, especially the fact that complexity has simply been shifted to other places. It's much easier for me to read a systemd unit file than it was for me to read a complex set of init scripts. The problem is that when I have 10 or 20 units with interlocking dependencies, it's an absolute nightmare to try and untangle because I have to jump between so many different files. Having to specify After=X and Requires=X seems like a prime example of optimizing for the 1% usecase. 99% of the time, when A must start after B, it's because A requires B. Having to specify both is something that is easily and often missed, and a hard problem to track down if you didn't read the documentation carefully. There's also the problem that it seems to be metastasizing and taking other parts of the OS. The longstanding /var/log/messages is now replaced by journalctl tool, and the logs are actually in binary. The venerable "hostname" command is now "hostnamectl" because it must go through systemd. They have also replaced chroot with systemd-nspawn, and made it so you cannot even use systemd while in a normal chroot!

vacri3• 11 years ago

$0.02: journalctl isn't even tab-completable until you've nearly finished typing it. :)

ibotty• 11 years ago

depends on what you have in your path.

erikb2• 11 years ago

Have you tried the systemd tooling that graphs depencies for you? I haven't worked on that area yet, but my colleagues made heavy use of that and actually could resolve problems with these quite fast thanks to the diagrams that systemd can create for you.

EmanueleAina27• 11 years ago

The problem is that when I have 10 or 20 units with interlocking dependencies, it's an absolute nightmare to try and untangle because I have to jump between so many different files

How that's different from sysvinit?

The longstanding /var/log/messages is now replaced by journalctl tool, and the logs are actually in binary.

`journalctl` prints plain text on its stdout, so you can think of it as a glorified `cat` if you don't want to use its extra features. On the contrary, if one needs its extra features on plain text log files one usually resolves to using fragile parsing hacks.

The venerable "hostname" command is now "hostnamectl" because it must go through systemd

They do different things, `hostname` only reads the current hostname. `hostnamectl` is able to set the hostname, standardizing what used to be a per-distribution custom mechanism. Despite being shipped with systemd (to make sure everyone can rely on it) it doesn't actually depend on systemd and in no way goes through it.

They have also replaced chroot with systemd-nspawn, and made it so you cannot even use systemd while in a normal chroot.

`chroot`'s purpose has never been the ability to run an init system inside it, no matter if it is sysvinit or systemd. On the other hand `systemd-nspawn` creates isolated containers, I'd be surprised you can't run sysvinit inside it.

Again, you're comparing apples to whole orange trees as the systemd utilities do much more than the traditional ones you mentioned.

keypusher2• 11 years ago

How that's different from sysvinit?

In sysv I can just make a list of startup tasks. A -> B -> C -> D. Easy. 10 things that must start in specific order? No problem, 10 things in list. There is no easy way to do this in systemd. I must create 10 different unit files, and then create a bunch of different dependencies (A depends on B and C, B depends only on D, can C then start at same time as D?, etc). When those dependencies change, I have to go look through a whole bunch of files or look at a graph tool to figure out what is even going on.

They do different things, `hostname` only reads the current hostname.

That's... not true. From the first line of the hostname manpage: "Hostname is used to display the system's DNS name, and to display or set its hostname or NIS domain name."

chroot`'s purpose has never been the ability to run an init system inside it

That's fine, but it worked under sysv and now it doesn't. When upgrading or working with different install roots, using chroot is common. Now I can't modify services during that time, unless I migrate everything over to use nspawn.

the systemd utilities do much more than the traditional ones you mentioned

If I lived in a blue sky world, where I was fresh installing one new server with a simple set of services that I had built in systemd, I think I be quite happy with it. The problem is that I have thousands of servers, and a decade or more of accumulated infrastructure that relies on those traditional tools. Then systemd comes barging in and breaks a bunch of stuff, and that is frustrating. I'm not saying I hate systemd, in fact I agree that many of the new tools are quite nice. But it's kind of like the loud hipster that shows up at your party and starts talking shit about the music you are playing, and making fun of your choice in beer. Maybe my music does suck, and maybe these clothes aren't cool anymore, but we were sitting around having a fine time until he came around.

ghshephard• 11 years ago

There is something epically beautiful about OpenBSD's consistency over the last 10 years. I swear that a 2004 OpenBSD sysadmin would be close to 100% efficient with a circa 2015 OpenBSD system, despite the absolutely massive number of improvements and changes that have been made with the operating system.

Once systemd starts to get traction, it's cognitive overhead is going to become a real problem for sysadmins who want to be able to model the entire boot process and run-state in their head.

For better or worse (I honestly don't know which, btw), you are going to have to start to trust the operating system to manage its boot process/run-state correctly rather than having it be a direct implementation of your mental model. (I would argue other *nix Operating systems, like OS X, have, to some degree, been doing that successfully for a while, so it's certainly possible - just not clear whether you want that type of behavior on your server systems. I guess time will tell.)

stephen_g• 11 years ago

I'm not a professional sysadmin, but as a developer with a bunch of virtual servers systemd has only made things far easier.

Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...

Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.

empressplay3• 11 years ago

...and that's awesome if you don't need 99.9% uptime and thus need to have absolute faith in your system's stability. However, for mission-critical applications that do it's hard to argue that systemd is a better solution than writing all your scripts by hand. Unless you want to be able to blame systemd if it all goes pear-shaped but I doubt your boss will buy it =)

anko• 11 years ago

Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...

This.

I'm not saying systemd is the best answer, but I really think sysadmins could learn a thing or two from developers. Nearly every custom init script I've ever seen has bugs, and isn't portable between systems. Where are the unit tests? Code reuse? Focus on maintainability/readability?

Shell scripting used to be amazing, but as a language these days it's behind the times.

pnathan2• 11 years ago

Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...

I've always found init scripts straightforward to read and write, even though I don't develop them often.

I don't care for the upstart/systemd/OSX approaches. I don't want to digest another manual to do what I can do in bash easily.

yellowapple20• 11 years ago

Professional sysadmin (and developer) here :)

Writing init scripts used to be a real pain

In the GNU/Linux world, yep, this is the case. Lots of copypasted boilerplate that becomes a nightmare to debug.

In the BSD world (at least with OpenBSD, where I'm most experienced), this isn't the case at all; OpenBSD ships with /etc/rc.d/rc.subr, which one can (and should) source into one's initscripts to make life much easier than even (in my experience) systemd's unit files. Set the $daemon variable, source in rc.subr, call rc_cmd, done. Sure, you might want/need to read a couple manpages, but this is the case with systemd, too.

Back to the GNU/Linux world, the complexity of SysVInit is not a hard dependency of using shell scripts for daemon management; there's very little stopping a GNU/Linux system from adopting a BSD-style rc system, rc.subr, etc. and making life significantly easier for sysadmins. Alternately, there's very little stopping a GNU/Linux system from implementing shell scripts in something significantly less horrendous to use, such as an interpreted scripting language (like Perl or Python or Ruby). SysVInit and BSD-style rc-based init both have a distinct lack of hard requirements on how initscripts are implemented, so long as said scripts are valid executables.

Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.

Would you not want to be able to model the startup process and run-state of the software you write in your head? Why should the underlying operating system be any different? If you have to rely upon external tools just to be able to reason about the execution flow of something - your server, your program, your recipe for lamb vindaloo, whatever - chances are it's way too complicated and needs significant refactoring.

empressplay3• 11 years ago

Except that there's no reason for systemd's model to "get traction" outside of desktop-focussed Linux distributions, since a "better user experience" is it's current raison d'etre.

So no, I shouldn't need to have to "learn to trust the operating system / systemd" on my servers -- it has no reason to be there!

justincormack• 11 years ago

The model where services do not daemonize and write a pid to a file is actually hugely better for servers. Most of the rest of it is not useful.

EmanueleAina27• 11 years ago

Many systemd features are explicitly tailored for server workload (eg. resource management), and a particular effort is being put over container-based deployments.

CoreOS has been quite successful in basing their own infrastructure on systemd and with rkt they are moving to use systemd even more.

devnonymous• 11 years ago

One of the more sensible ^rants^ I've read against systemd (it is KISS for some weirdly convoluted definition of KISS). Too bad these naysayers are not/will not be taken seriously.

vacri3• 11 years ago

The naysayers are taken seriously, but they're in the decision-making minority. Linux distro maintainers wanted to move on from SysV Init, and systemd looked promising. Just because the naysayers didn't get their way (and on some distros, they did) doesn't mean that their criticisms aren't noted.

thaumaturgy3• 11 years ago

Which distros? Slackware is the only near-mainstream Linux distro I know of that is even possibly not moving immediately to systemd, and even then, the Slackware folks have said that once it becomes too difficult to maintain upstream packages, they'll switch too. They're just being pragmatic about it.

VLM2• 11 years ago

"Linux distro maintainers wanted to move on from SysV Init"

LOL no. distro maintainers wanted to ship gnome, which was product tied to systemd. No systemd, no gnome, that simple.

sevensor• 11 years ago

Agreed --- this is the first rant against systemd I've seen that didn't talk about Lennart at all, and the first one from which I've ever learned anything. It's a great example of how people ought to argue on the internet. While I find that systemd is great for my purposes, there's plenty of room on the internet for Steven and the BSDs.

swhipple• 11 years ago

The author mentions that he believes the biggest problem of systemd is repeating some mistakes of SysV init, but I think it's actually pretty important that familiarity is preserved to help people upgrade. For example, the author mentions babysitting the users (systemctl edit) and config/daemon reloading as things that could be made automatic, I'm assuming with e.g. inotify change events.

But you often want some confidence that the change you're making to your system is what you expect, and we've had them all along:

- visudo(8) sanity checks, for atomicity and syntax, to help prevent you from bricking your authorization

- an explicit SIGHUP to reload the configuration file for your daemon, after you've checked the config, rather than accidentally saving your file while you're editing it and bringing down your daemon.

You could expect everyone to adopt new practices for editing the configurations, but the behavior would be very surprising to anyone used to the old way.

bluejekyll• 11 years ago

What I think your getting at is that essentially you need a mechanism to commit your changes before the thing that's reading those changes receives them. In essence, an atomic operation for the change. Completely agree.

pjc504• 11 years ago

I'm hearing "we need to build a version control system or a transactional database into our init daemon" here :)

stormbrew• 11 years ago

So I have not yet had to use systemd in any serious way, so I may be misinterpreting here, but I think this:

... and config/daemon reloading as things that could be made automatic, I'm assuming with e.g. inotify change events.

is a misinterpretation of the author's complaint. It sounds like with systemd, if you tell it to restart a service it will continue to use the old definition until you run a separate command to 'reload' it (and, presumably, all other configurations). On top of that, it indicates its knowledge that you've changed the file by telling you to run the aforementioned reload command.

No inotify seems to be involved in what he wants, which is that the reload command use the new configuration. Hell, be clever and use the old configuration's stop mechanism and the new one's start mechanism or complain if the restart mechanism differs or something, but a message telling you you didn't do something you almost always want to do is a pretty obvious smell for a UI problem.

I deeply dislike the similar behaviour of upstart, fwiw. In order to reload an upstart init script you have to stop and start it, restart uses the old one. I have spent annoying amounts of time wondering why it was doing it wrong.

XorNot2• 11 years ago

Because systemd is event-driven, you can't actually be sure the service file won't be re-used in the running configuration - or that it won't trigger other service reloads/starts/stops due to its dependency specifications. Ergo systemd requires an explicit reload to commit the state of the unit files once you're satisfied they are as intended.

This is very important when you consider that unit files can kill other units on their conditions - I've used this feature in particular to make sure nginx goes down if a mountpoint becomes unavailable or the servicing process dies (and systemd can't restart it). This is extremely useful because it means a load balancer won't send requests to a webserver in an improper state. But it also means that while editing those config files, I wouldn't want systemd auto-reloading everytime I saved.

invernomut0• 11 years ago

You can also add 'crontab -e' and much other examples of commands that wrap an editor call already exising in linux userspace.

jveld• 11 years ago

It got the job done by making the 99% majority of use cases more difficult in order to make the 1% minority simpler. This is a design pattern I think is being repeated in systemd.

I kinda had the opposite impression. I first encountered systemd when I started messing around with arch for a couple little vps projects. At the time I was doing sysadmin that involved both Ubuntu and CentOS, and I definitely found day-to-day tasks easier with systemd (better built in monitoring, consistent syntax, etc). For simple cases, writing a unit file sucks a lot less than writing an initscript. The warts in systemd only started to irritate me when I was first setting up my personal machines (one laptop, one tower) to run arch [1]. Even then, the bloat in the design irked me more than actually using it. So, I'm in favor of systemd, but not thrilled (dbus everywhere is annoying).

Then again, I don't maintain any packages with strange and/or complex daemons. If anybody's had a really hard time getting their stuff to work with systemd, I'd be interested to hear stories.

1: Not completely true. I want my damn log files back.

thaumaturgy3• 11 years ago

This is why I suspect that most of the divisions in the sysadmin community regarding systemd are generational. For people (like me!) who have been writing init scripts forever and can do it in their sleep, they'd prefer the flexibility and familiarity of init scripts to a new system that at first glance appears more limited.

For people who are less experien^H^H^H^H^H^H^H^Hcrusty :-), init scripts look like a horrifying way to manage a startup process, and just about anything -- including Yet Another Config File Format -- looks preferable.

I think this is also why so little progress is being made on resolving conflicts over systemd. There really isn't a whole lot of room for compromise; one group simply wants things to stay more or less the same (or evolve incrementally), the other group simply wants to throw it all out and start over.

I bet there would've been a lot less argument and vitriol about systemd if Debian hadn't made the decision to default to it, basically leaving all the crusty sysadmins homeless. Once Debian switched, there was a near guarantee that all of Linux was going to move to systemd. That suits one group just fine, another one not so much.

On the plus side, the BSDs should be seeing a really nice upswell of installations over the next couple of years, which is great, because they've been quietly building some pretty great operating systems that deserve more love.

chousuke2• 11 years ago

It's perfectly possible to run scripts with systemd unit files, so there's no real "flexibility" lost anywhere. The primary improvement is that systemd handles by default most of what used to be boiler-plate in every script, on top of calculating dependencies automatically. Then you get integrated process supervisor capabilities so that the init system actually knows whether services are running or not.

Speed was certainly a major motivator as well, considering the increasing popularity of containers and on-demand virtual machines.

To me it seems that "gradual evolution" was exactly what they did, and it wasn't really going anywhere. Lennart came up with systemd and convinced enough people that it's worth the pain of migration to get a clean start.

pppp• 11 years ago

I wouldn't say the init scripts on OpenBSD are complex:

# cat smtpd

#!/bin/sh # # $OpenBSD: smtpd,v 1.4 2012/10/26 06:51:59 ajacoutot Exp $

daemon="/usr/sbin/smtpd"

. /etc/rc.d/rc.subr

pexp="smtpd: \[priv\]"

rc_reload=NO

rc_cmd $1

EmanueleAina27• 11 years ago

You can still use plain shell-based LSB-compliant init scripts just fine, which is what Debian is still doing as they didn't want to have flag days like all the other distributions who switched.

I don't see why you should feel homeless.

yellowapple20• 11 years ago

The nice thing about initscripts, though, is that you can get the best of both worlds by sourcing in common functions, as is the default and recommended practice on OpenBSD. For example, here's the initscript for dovecot on my (OpenBSD-running) mail server (with some comments and newlines omitted for brevity):

    daemon="/usr/local/sbin/dovecot"
    . /etc/rc.d/rc.subr
    rc_cmd $1

Pretty straightforward: you set $daemon to the path of your daemon's executable (in this case, dovecot), source in rc.subr, and call rc.subr's rc_cmd command with $1 (which represents a command, like 'start', 'stop', 'restart', etc.).

rc.subr, meanwhile, holds all the complexity one will ever likely need in an initscript, in the form of various shell functions encapsulating things like starting, stopping, etc. And if that's not enough for whatever reason, it's easy to add more functionality either to the individual initscript or to rc.subr, since it's shell scripts all the way down. In my experience, reasoning about this sucks a lot less than writing a systemd unit file, let alone an initscript on a system without such niceties.

digi_owl19• 11 years ago

That is the crazy part, systemd at its core is a reaction to sysv (and upstart).

In part part the reaction is to the boilerplate nature of sysv scripts, as they bring all their logic with them.

but if you glance outside the RH/Debian sphere you find the likes of Slackware, that has been using BSD style init for decades.

stoplight• 11 years ago

Am I missing something? The author states:

Try to find the answer to "which units are going to be started on next boot?".

Isn't this just `ls /etc/systemd/{user,system,network}//*` (bah HN formatting is messing this up)

Also the other states:

Speaking of which, why is systemctl daemon-reload even a thing?

Because configuration changes while things are running. Otherwise we wouldn't need apache2 reload, nginx reload, haproxy reload, etc. Also, according to the man page:

After the units have been edited, systemd configuration is reloaded (in a way that is equivalent to daemon-reload).

As for systemctl edit, it allows you to add diffs to the unit file. For example, let's say I am using redis and I want to ensure it always restarts (for whatever reason). Rather than manually editing the unit file (which will have issues if an upstream change happens), I can simply use systemctl edit to add a diff for my change. This way I can still maintain compatibility with the upstream unit file instead of having to deal with potential conflicts. It's not so much babysitting the user as it is a potentially better experience.

ploxiln2• 11 years ago

Isn't this just `ls /etc/systemd/{user,system,network}//*` (bah HN formatting is messing this up)

Nope.

There are many virtual units for which there are no files. One example is the on-demand getty units. There are two for "plymouth" boot-splash stuff (I see these attempted-and-failed units on archlinux where plymouth never exists). There's some autofs thing. There's auditd.

I'm actually not booted into linux right now so I'm struggling to remember the specifics... but if you also count all the simple mounts and devices which clutter up the output of "systemd --all" as "units which are not files in /etc/systemd" then the count is around 100.

INTPenis• 11 years ago

Couldn't/Shouldn't this be solved with some script that traces the dependencies of units?

I'm not familiar enough with systemd to understand how to trace dependencies and create a complete list of all units that will run at next boot but I expect this to be doable by someone with that expertise.

groks4• 11 years ago

    systemctl list-dependencies

belorn• 11 years ago

With System V and Debian, one added some new apache or bind configuration and then called invoke-rc.d daemon reload to inform the daemon to reread the configuration files.

With systemd, you do the exact same thing and then call systemctl reload daemon. Is the author making an argument here based some other distributions way of handling System V?

JdeBP9• 11 years ago

No. The author is not getting reloading the daemon's configuration mixed up with reloading systemd's configuration. You, however, are. The systemctl manual page warns about this mistake. See what it says under daemon-reload.

weland19• 11 years ago

I'm one of the people who jumped ship to OpenBSD in the last year or so. I think I first touched a Linux machine in 1997 or 1998. systemd has been part of the reasons I did that for precisely the same reason.

There is a lot of development in the ecosystem dominated by Freedesktop.org, Gnome and (to some degree) large commercial players like RedHat and SUSE. It's more efervescent than it's ever been. Fifteen years ago, I'd have been ecstatic to see so much development being done.

Yet for all this fierce code churning, there's absurdly little innovation. Even in systemd, there's a lot of focus on non-issues like booting without needing a shell, while still inventing service startup notification types because none of the gazillion startup notification types already existent can actually solve precisely the problem systemd claims it solves -- managing complex startup chains.

In the last three or four years, my Linux systems have become increasingly blackboxed, and consequently harder and harder to configure, let along debug when they stop working. Over the span of six months, a tool you just got to learn how to work with (mostly because the documentation seems to have been written by very bored and very verbose monkeys who had only a typewriter to pass the time with) becomes obsolete or is revamped to be unrecognizable. Which would be fine if that actually solved any problems. Instead, it replaces the old set of problems with a new, even more complicated set of problems that developers claim you shouldn't have, while insisting that the old problems were simply due to the old architecture and we'll have none of that now that we have this shiny new thing.

Ironically, systemd is actually one of the better pieces of software to come out of that ecosystem. No, seriously. It's decently documented and as long as you don't need anything more complex than "here, start this daemon after this other daemon started", it does its job.

If you want to peck at something, have a look at Polkit, where someone brilliantly figured out that the best way to configure policies is to write basically undebugable JavaScript code in .rules files.

anon3_2• 11 years ago

I'm one of the people who jumped ship to OpenBSD in the last year or so.

Ha. FreeBSD here.

And so long as you get the right hardware, I find BSD to be far simpler than linux distributions out there.

Ports? Beautifully elegant and simple. Configurable. Magnitudes easier than configuring rpm/debs.

Getting a package approved in debian? Alioth? No offense, the wiki, all these pages on the debian website - they have no concept they have of "Handbook", in the sense FreeBSD has. It's so adhoc, they're healthy, but they're in denial about how hard they make it for new people. By the time you've taken to time to learn Debian's dev warts, you're apart of the convent and no longer care about fixing the onboarding problems.

But to be fair. I have day to day gripes with FreeBSD just because we're short on the manpower Linux distributions have in excess:

On FreeBSD 10.1, chromium crashing easily, settings doesn't work. Firefox just clogs up and stops working. No skype (CURRENT will be in a better position to since epoll support has been added). No Dropbox (as of now, we don't have inotify, and kqueue may not be able to handle it, don't quote me on that).

I do my 90% of daily routines simpler and better on FreeBSD. If only Google / Mozilla (with all the resources they have) gave someone who could work to fix this stuff.

weland19• 11 years ago

I've occasionally used {Net|Free|Open}BSD over the past few years (in fact, I think my Jornada in the back of the closet still runs NetBSD). I kept sticking to Linux partly because I was using it at $work, partly because it was simple enough and I'm kindda lazy. But yeah:

I find BSD to be far simpler than linux distributions out there.

The way OpenBSD handles WiFi configuration is a good example of this. Want to connect to a wifi network? ifconfig <interface> <ESSID> <passphrase>. If you need some other fancy flags for those 0.1% of hotspots in the world, you can do those, too (also through ifconfig) but it's that simple.

In contrast, after everyone agreed that wpa_supplicant is probably what happened after Stalin's ethereal spirit had mad sex with Hitler's ethereal spirit and their ethereal baby was incarnated, what did we do to alleviate things in Linux land? Why, we wrote NetworkManager, of course! Conservatively, 90% of today's deployments of NetworkManager exist because no one can figure out how to set up wireless without it, and we put up with its crap because wicd is waiting for us at the other end.

Better yet, this is all nicely documented in the manpage. If you search for "wireless" in ifconfig's manpage, you find out all you need to know. Just for fun, compare that to man networkmanager.

On FreeBSD 10.1, chromium crashing easily, settings doesn't work. Firefox just clogs up and stops working. No skype (CURRENT will be in a better position to since epoll support has been added). No Dropbox (as of now, we don't have inotify, and kqueue may not be able to handle it, don't quote me on that).

I haven't had crashing things on OpenBSD, but yeah, a bunch of things don't work. Fortunately, they're things I don't use. If I ever need them -- OH THE HORROR AND THE BLASPHEMY -- I have a Windows laptop, which for all intents and purposes is by now about as closed and as hard to wrestle with as Fedora, but at least it doesn't hopelessly break every six months. There are Linux-only things that I sometimes need because at some point between 2003 and 2013, "portable" ended up meaning "we can probably work if you have bash in a non-standard location but that's about it" (eh, Yocto?) for which a cheap VPS is more than enough. When it breaks, I don't have to spend days trying to find out exactly what part of PolicyKit broke or where my devices are mounted this week -- I just wipe everything and reinstall.

ank_the_elder• 11 years ago

I've used the crux linux distro (https://crux.nu) because it has a similar, BSD-like package manager, it's very clean and has a great community.

mavhc• 11 years ago

I used FreeBSD for a while because it had ZFS, and found Ports confusing and fragile. Maybe I was doing it wrong, but the handbook wasn't much use: "here's 4 different ways to do it, we're not saying which is a good idea". Every upgrade took ages, required reading a load of documents, and broke something.

There didn't seem to be an equivalent of Debian's Stable, everything was apparently the latest release.

justwannasing2• 11 years ago

To be fair, chromium crashing is due to the increased complexity of chromium and nothing to do with FreeBSD as the maintainers have noted in their comments. They have found it increasingly difficult over the past couple of years to keep up with it though more manpower would help I'm sure. But such issues are fixed within a couple of weeks or days. the issue you speak of has to do with javascript in GMail and settings and how chromium interacts with that; not chromium directly.

I don't have the issue with Firefox, and never had any issue using the latest hardware, and I don't think the lack of Skype is FreeBSD's fault either but I forget.

pjc504• 11 years ago

Polkit is a nightmare. And I agree with you on the churn. There was a time when Linux administration changed less frequently than Windows, round about the 95-XP-Vista era.

groks4• 11 years ago

http://smcv.pseudorandom.co.uk/2015/why_polkit/

digi_owl19• 11 years ago

There is a lot of development in the ecosystem dominated by Freedesktop.org, Gnome and (to some degree) large commercial players like RedHat and SUSE.

Yeah, i've been thinking the same recently.

The current development above the kernel seems to either be from the desktop down, or from the point of view of managing 1000s of headless instances in a "container" farm.

This with a side-order of Windows-isms demanded from corporate/government clients looking for a way to get away from MS while retaining their centrally controlled fine grained access control.

octref2• 11 years ago

This page's horrible readability is a perfect reflection of the kind of folks that dislike systemd.

Yeah, newcomers to the Linux land without the expertise you guys acquired through 10 years of using Linux, would apparently appreciate how easy it is to manage a server by invoking some lengthy commands they learned through reading the often overly-dense manual instead of `systemctl start/stop x`.

Nowadays some of those so-called "hackers" tend to ignore the part of work that makes the user experience better. They claim that they want simplicity and avoid bloat but most of the times they are just being lazy and not sympathetic to their users.

I'm not saying that things should be dumbed down to clicking GUI buttons on Linux. But if you want your program to become mainstream you need to think about providing good and practical docs and making your program easier to use, even for text-based programs.

Accept that most people would like Bootstrap'ed sites more than yours:

  body
  {
    background-color: #080808;
    color: #ffff70;
    font-family: sans;
  }

That's why I opt for Ubuntu 15.04 instead of 14.04 LTS on my VPS, since 15.04 has systemd.

cthalupa• 11 years ago

Are we really using website design choices as a valid target to discredit someone's opinion on the technical merit of an init system?

Can I use Lennart's inability to buy a properly sized polo as a reason why systemd is bad?

Yeah. The guy has an ugly website. But he has specific points about mistakes that systemd didn't learn from and is repeating. Did he list 'it's too user friendly' as one of those mistakes? In fact, one of his core arguments is that there will actually be an increase in complexity and reduction in user friendliness due to design choices that have been made.

octref2• 11 years ago

I'm saying that, if people keep responding to those less-experienced Linux users who complain init scripts too complex by saying "Quite. Deal with it." like this author (http://www.steven-mcdonald.id.au/ugly.shtml), I fail to see how systemd is gonna fail.

My comment is not about design choices but attitude and sympathy to users. This is not intended as an ad-hominem.

The author seems to think systemd "babysits" its users, but as a user I find systemd "cares" about its users.

tomgg• 11 years ago

1. The readability and typography are (to me) stellar. Would that more websites present information that clearly.

2. The author doesn't dislike systemd.

xyby• 11 years ago

This page's horrible readability

It might not look fancy, but the readability is way better then on most sites these days. It loads fast, it has enough contrast, there are no distracting elements like images or animations that make it hard to focus on the content, scroll works as expected, the content is not split into useless pages, it does not make my fan spin up because some elements use insane amounts of computation power, content is not squeezed into some tiny box as it often is on other websites...

If the average website would let me read as comfortably as this one, that would be a huge step forward.

DanBC• 11 years ago

Fwiw I like the website. I would have toned down the yellow. And I would probably have had dark on light.

But it's really great to have a page that loads nearly instantly - the ratio of information provided : loading time is very good.

Other pages take an age to load (and Chrome on iOS has an annoying "feature" where the address can't be selected and copied until the page has almost finished loading).

wooger• 11 years ago

Agree, it's a pleasure to use a site that isn't burdened by tons of ads, tracking scripts, and tries to re-implement scrolling (badly) if you visit from a mobile device.

See theverge.com for an example of a site that doesn't even scroll smooothly on my android phone.

TickleSteve• 11 years ago

Readability != style & aesthetically pleasing.

aaronem2• 11 years ago

Some complexity is essential, and not everything is easy. Pretending otherwise is a great way to end up sorry.

pjmlp• 11 years ago

I don't care about systemd, as I want to use OSes that embrace modern architectures, unikernels, containers, safer programming languages, improved UI stacks,....

This rant is a good example how luddite users rather stay stuck in a UNIX System V world with their green phosphor VT100 terminals.

IshKebab• 11 years ago

Well said. This reads like someone complaining about the metric system because you can't easily divide metres into twelfths.

Those kinds of people will always exist - they've spent many hours learning it the good old hard way, and I'm damned if you kids are going to make it any easier! You lazy so-and-sos.

digi_owl19• 11 years ago

Thing is, Linux used to be flexible. This because of some stuff it inherited from Unix about user space being made out of loosely coupled parts with solidified interfaces between them.

Meaning that it could accomodate both the latest and "greatest" coming out of the labs, and the tried and tested stuff that had been sitting in production for a decade.

systemd instead gloops it all into a lightly coupled shoggoth of a project that is chasing the future come hell or high water.

xroche• 11 years ago

nearly everyone (or at least the most vocal crowd) seems to either love it or hate it

"At least the most vocal crowd". Which means that most people DO NOT CARE, and the systemd war is beyond ridiculous.

As Linus said, > "I don't actually have any particularly strong opinions on systemd itself."

jfindley• 11 years ago

The systemd war is probably the best thing that could have happened to ensure widespread adoption of systemd. It allowed the supporters to lump anyone with any concerns or objections in with the frothing-at-the-mouth haters, and completely ignore them.

I don't hate systemd. I like what it's trying to do, and I don't hate the big-picture implementation either. But I, and many others, have a bunch of concerns around some specific details of the system, that got ignored and swept under the rug in the tide of vitriol between the two sides of the "war".

If the discussion had been more civil, maybe the more rational objections might have got heard, and we might have been able to pressure Lennart and Kay into not rolling their own, un-reviewed, crypto for journald[0]. We might have been able to avoid some of the feature-creep[1] that seems to have grown out of a NIH-mentality, and a better relationship with the community might have made the core developers more willing to play nicely with others[2][3].

0: http://lwn.net/Articles/512895/ 1: http://www.freedesktop.org/software/systemd/man/systemd-reso... 2: https://bugs.freedesktop.org/show_bug.cgi?id=76935 3: https://bugzilla.redhat.com/show_bug.cgi?id=1116651

EmanueleAina27• 11 years ago

Why is resolved feature-creep? It's rather nice to have working name resolution out of the box when coupling it with networkd and firing up a container.

I mean, it's not my standard usecase in this moment of my life, but I see how it can be useful.

(Also, your last link reports that "This bug is for livecd-tools which has been fixed", I don't know what's the point you're trying to make.)

eleitl• 11 years ago

Pretty much the same reason why I'm slowly moving to FreeBSD. OpenBSD is another possibility.

stefantalpalaru2• 11 years ago

Why would you punish yourself with various *BSDs when you can have OpenRC and more advanced ports on Gentoo? Do people really associate systemd, Gnome and other nuisances with Linux?

rockdoe2• 11 years ago

Given that it's the default and sometimes only working option for many of the most popular Linux distros: yes.

yellowapple20• 11 years ago

Given also that systemd is Linux-only due to its reliance on cgroups: yes.

geggam• 11 years ago

Punish ? .... BSD is extremely user friendly. It just seems to be picky at who it calls friends. They ONLY reason I still use Linux anywhere is because I have to keep my skills for it fresh since its where I make a living.

The metric I like to use for adopting new processes / services is pretty simple. I have _____ problem. What tool solves that problem for me.

In my case systemd solves nothing and creates more.

digi_owl19• 11 years ago

Punish ? .... BSD is extremely user friendly. It just seems to be picky at who it calls friends.

I seem to recall the same line being used about (GNU/)Linux at one point.

BFay• 11 years ago

Don't you have to build every package from source with Gentoo? I can see where that would be useful, but isn't it really time-consuming (or punishing, if you prefer that word)?

stefantalpalaru2• 11 years ago

It wasn't a deterrent when I had a Pentium 3 at 600 MHz, it sure isn't now when I have a FX-8320E at 4.5 GHz.

I actually prefer from-source package installs because I get a higher degree of control over configuration options.

vezzy-fnord3• 11 years ago

To all those who complain about unreadable initscripts, please see: http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/ru...

davexunit• 11 years ago

Some of the complaints resonate with me. In particular, the limited power of systemd's unit file DSL is why I think it's better to allow services to be written using a general purpose programming language. GNU dmd, while still not developed enough to do cool things like socket activation, gives the user the full power of Scheme to configure their system services as they like. I run it as PID 1 on two of my machines, and I also run it as an unprivileged user to manage my gpg agent, emacs daemon, offlineimap, mpd, development web servers, etc. on all of the computers I use, including at work. I think people that aren't happy with systemd, that are also interested in rolling up their sleeves and hacking on an init system, should give dmd some love.

https://gnu.org/s/dmd

Tiksi2• 11 years ago

About a year ago, we started switching over new servers to systemd instead of openrc at work. I knew sysv/openrc/bsdinit and I liked them so for the first month or two I utterly despised systemd. (Plus as a sysadmin it seems like I have to hate it or risk losing my linux card) I just wanted my damn scripts back instead of dealing with the .ini-inspired config file crap. If I had a choice at the time, I would've been willing to move to busybox-init just to get away from systemd. Had I spent less time complaining to my coworkers about how much it sucked, I probably would've written this post myself. But little by little it grew on me.

Oh, so if an nfs mount fails on boot, it's not going to just sit there and hang? I guess that's pretty cool, saves me from having to throw a console on it while muttering under my breath. And it didn't start nginx either, so it's not going to serve a half broken site and tell the loadbalancer all is good? Maybe it's not all that bad. No no, what am I thinking, this is still crap, I could've written all this in to the init scripts (but I didn't).

This client's service is irritatingly unreliable, maybe I should write a wrapper to supervise it. Someone must have done something like this for syst... Restart=OnFailure... well that's nifty.

Ugh, monitoring daemon ate up all the memory again, why can't the vendor get their shit together or the client switch to something that isn't useless. Guess I'll google some ideas for a watchdog script. Hmm, looks like I can just throw MemoryLimit and Restart=onfailure in the unit? No way it's that easy. (it was)

After a couple months of this, systemd actually started to seem useful, I even started having a little bit of dread when hopping on an older server and having to deal with openrc. All of the complaints here really seem off base to me now that I've had a chunk of time to figure it out. Init scripts pissed me off when I didn't have much experience with them too. Now, thats not to say I like everything about the systemd switch. netctl is utterly worthless for anything other than a static ipv4 address (and reenable, really?). I still forward all my logs to syslog, because I prefer text files (and all the awk/sed/grep/cut/sort/column/etc goodness that comes with them) for anything more than running `systemctl status <service>` and seeing that I accidentally the config file for something.

I also don't buy the whole "It should be on desktops, not servers" because I don't see much of an advantage for desktops, other than maybe the trivially faster boots. Wifi is a pain in the ass with netctl and netctl doesn't seem play well with anything else. But for servers, I've found it to be pretty awesome.

To address a few things straight from the article that I disagree with:

At least one Debian package (memcached) is already invoking a wrapper script from ExecStart.

This is exactly why I don't understand the whole argument around it being inflexible. If you need to call a script that does some setup that's incompatible with systemd, call the script? It's no different from helper scripts that many packages used with init scripts. If you still want to do some checks and call start-stop-daemon, go right ahead, nothing is stopping you. It seems pretty strange to consider it a fault that you can't use a bash script, and then complain that you'll have to use a bash script.

Here, let me helpfully tell you that you should run systemctl daemon-reload, but perform the action you requested using the old configuration anyway.

This really seems like a trivial complaint. I guess you could always alias systemctl="systemctl daemon-reload; systemctl" if it's that much of an issue. Personally it's saved me after an accidental :wq (damn that muscle memory). "File has changed on disk" uhhh it shouldn't have. crap. I guess it's a matter of preference. Also I didn't even know systemctl edit was a thing. That does seem kinda pointless, but it also seems kinda pointless to be bent out of shape over it. Do you also cat `* * * * * command >> /var/spool/cron/crontabs/$USER` because crontab -e is babysitting you? (I used to do that, but because e and r are too close together and reconstructing a crontab from logs is painful. Now I just alias crontab="crontab -l; crontab")

Now, systemd does rely heavily on D-Bus

I've never run into issues with it, but I do agree with you on that point from a philosophical standpoint.

And yes, you can avoid the whole issue just by masking any units you don't want to start, but that's missing the point.

The only point I see is that it's different. Of course whatever you're used to is going to be more effecient to you than something new.

I believe it creates more problems than it solves. Sadly, many of those problems will take years to become apparent, by which time all the major Linux distributions will have fully integrated it.

Well it definitely seems to solve more problems for me than it creates, and saying those problems will only be apparent years down the road is not only impossible to argue against, it's a hand wavy cop out of "Trust me, its just bad! Because reasons!" However this does touch on what is probably my biggest gripe with systemd, and that is that ALL the big distros seem to be switching. Granted I don't mind from a having-to-use-it angle, since I personally prefer it and would probably stick with distros that do anyways, but it does give me a certain unease, it may be my choice but it shouldn't be the only one, and I'd have the same problem if it was sysv that everyone switched to.

All in all, these complaints really just seem like complaining that it's different. And it is different, but that doesn't necessarily make it better or worse, just takes time to get used to.

digi_owl19• 11 years ago

You either must have gotten in late, or been exceedingly lucky, as failing NFS mounts is one of those things systemd downright hates.

People have had whole boxes fail to come up because of a NFS mount that didn't work, and therefore the unit file didn't come up roses, and so systemd decided to abandon ship and present the admin with a borked minimal console.

One early fix, a brute force 30 second kill timer on every unit file.

Went over well with one Fedora Alpha (lucky for them) where said timer tripped on the post distro upgrade house cleaning unit.

exelius3• 11 years ago

This is the first anti-systemd article that I've read that focuses exclusively on the technical issues with it (well, mostly: the last one kind of gets touchy on the politics). And this article is a good summarization of them.

Linux in the server world is rapidly turning into a thin platform hosting containers (docker, kubernetes, etc) of "fat" Linux distros. Systemd makes a lot more sense if you consider it in that context: a container can use systemd to have the host start a monitoring process whenever any processes are executed, etc. Sure, this probably could have been grafted on to sysvinit, but it would have been an ugly hack just like every other thing that's been grafted on to sysvinit.

Linux everywhere else but the server world is all about low power usage and tight integration with custom hardware. Embedded devices, mobile phones, tablets, wearables, etc. All of these use cases demand an init system that is aware of and able to manage the system power state as a gradient, not a binary value. Again, this could probably be hacked on to sysvinit or the kernel itself, but would require big kernel changes and likely a fork for "Mobile Linux" and "Server Linux" as the market begins to demand power-aware schedulers, memory mappers, etc.

If the computing world were to remain in a paradigm where we run small physical or virtual servers dedicated to tasks, then sysvinit would be fine. But that's not where we are today nor where we will be, and sysvinit simply isn't flexible enough for a lot of the use cases where Linux is seen today. The *BSDs exist if you want a rock-solid and simple x86 Unix system for hosting traditional applications in a client-server setup; but if you want a flexible platform for building arbitrary products, Linux is the way to go.

jeffreyrogers• 11 years ago

Thanks, I think your perspective on the different use cases for Linux vs *BSD were helpful and something I hadn't seen all in one place before.

monksy• 11 years ago

Thats one responsive site!

jingo• 11 years ago

"It babysits its users"

mm_aa• 11 years ago

It seems that the author completely disregards the modern tendency to treat boxes as commodities. While pondering a "what's going to start on this particular machine" question is interesting in the "old world", it is hardly a question anyone asks when your fleet starts to be measured in tens of thousands of cloud instances.

digi_owl19• 11 years ago

No, his issue is that because others choose to do so he has to do so because the fad of the year is tramping all over the alternatives.

tfft• 11 years ago

I love the valid html/css icons at the bottom while the whole page is just a yellow text on a black background.

Why I dislike systemd

Related Stories