What should we expect from a modern init system?

https://plus.google.com/u/0/+LennartPoetteringTheOneAndOnly/posts/ip8e1DqJdxT

by martius • 13 years ago

62 95 13 years ago

plus.google.com

samworm• 13 years ago

I personally prefer systemd to upstart. I just hope that distros (in this case Debian) stick to a technical review and don't get drawn into ad hominem arguments.

For many people it seems the answer to the question posed in the post title is "I would expect it to not be designed/authored by the man behind pulseaudio".

martius3• 13 years ago

One argument that is on the edge: Lennart often takes controversial decisions, and in the case of systemd, he refuses portability patches that would allow systemd to run on other unixes than Linux (BSD, Hurd, etc).

Since debian is supposed to be universal (hence, support alternative Unix kernels), choosing systemd barely means that they will have to drop this trait of the distribution.

krakensden• 13 years ago

I suspect that if someone actually did the work to port systemd, upstream would be nicer.

I have mixed feelings about the BSD & Hurd ports. "You shall not crucify Debian Linux upon a cross of Hurd" and all that.

fournm• 13 years ago

Upstart also has the issue of being non-portable as of right now, so either way a porting effort has to be actually undertaken (even if Upstart is saying that it's planned)--And Debian is adopting large amounts of systemd for other systems.

Lennart hate is fine, but let systemd stand on its technical merits (or lack thereof, if you really feel that way).

theophrastus• 13 years ago

There is an observed situation in evolutionary ecology which the response to Lennart rather reminds me of: the case of the overwhelming singular adaptive advantage. An ecological situation arises which gives one subspecies a huge, but temporary, advantage. So huge that all the other subspecies are competed to extinction. But that advantage was an advantage in only one aspect of survival and now there's a monoculture (which we all know is more fragile against future threats). If there's a latent, (or perhaps overt?), Lennart-phobia, it's that he possesses that one temporary huge advantage which will lead us to a monoculture? ...or not, this is just a curious analogy [shrug]

bkor7• 13 years ago

Various features that systemd exposes have no equivalent in other kernels. This makes it impossible. He also explained various times why he refuses non-complete portability patches. This because you get more ifdefs and those are difficult to test/guarantee.

He and Kay claimed that kdbus couldn't be used for Binder as well. Greg thought it was possible. Only after writing kdbus he came to the conclusion that Lennart+Kay had it right.

Suggest to write a complete patch to make systemd work on FreeBSD. I assume you'll discover Lennart is right :-P

Shish2k2• 13 years ago

I would expect it to not be designed/authored by the man behind pulseaudio

Having used a bunch of Lennart's software, I'm generally coming to the conclusion that he's better at ideas than code -- but once his projects take off and the bugs get fixed, the end result is significantly better than what we had before.

(Using pulseaudio as a specific example, per-app volume controls and easily mixing and matching sound sources and playback devices; seamlessly switching from speakers to headphones to HDMI to a sound server on another PC entirely, etc -- features that are too high-level for the kernel and too low-level for applications, and thus can only be brought into existence by a fairly large change in the audio stack)

nitrogen• 13 years ago

I would prefer a parallelized dependency-based init system over an event-triggered system. I want to be able to say "start openssh" and have networking start with it automatically, for example, like Gentoo used to have (or still does?).

Tuna-Fish• 13 years ago

systemd does exactly that.

anaphor• 13 years ago

Not exactly. It does do stuff concurrently, but not always in "parallel" since it may only have 1 core available or not get scheduled across multiple cores. I disagree with that particular wording of the systemd description because it perpetuates the concurrency == parallelism misunderstanding. Not that I don't like systemd, I've been using it for over a year and haven't had any issues, and its concurrent model does seem to improve boot times somewhat and there's no more figuring out how to order your daemons since it handles that for you.

mverwijs7• 13 years ago

I would prefer to do my own thinking, not have the computer try to outsmart me. It can't, and it always hurts my sleepcycles when it tries.

Groxx• 13 years ago

Not sure how this applies. I assume you automate some things that you could simply type in by hand every single time, why not this?

bkor7• 13 years ago

My smart phone automatically silences from the Bluetooth speakers when I plug in my headphones. When I'm listening to my headphones and remove them, the music automatically pauzes. It thinks for me, and I think it is great!

I can also gives loads of similar examples for just a computer, but above behaviour is something I really appreciate every day.

forgottenpass• 13 years ago

You already have an init system that does some level of "thinking" for you. You had to learn how that one works, you just don't want to learn a new one.

__david__2• 13 years ago

I know! I still can't stand that my computer just boots when I turn it on. I liked it when I had to input my boot code in binary with the switches on the front.

Also, I refuse to use "make" because I prefer typing long and complicated gcc lines and working out all the header file dependencies in my head. Why on earth would I trust my computer to do that for me?

brokenparser3• 13 years ago

I expect an init system to be an init system, not to also be pm-utils, (x)inetd, acpid, syslogd, watchdog, cron and atd. I also expect it to have configuration files I can change, not to have everything symlinked to /usr. (If I can even find the right configuration file in that mess of directories, that is.)

antihero2• 13 years ago

It's quite trivial to override services - you just cp /usr/lib/systemd/system/myservice to /etc/systemd/system/myservice and edit it there.

This has the advantage of not having you screw up the actual package managed files, too.

rodgerd5• 13 years ago

No, you don't copy it! Source it ( . /usr/lib/systemd/system/myservice at the head of /etc/systemd/system/myservice ) then override the lines you want to change. Much easier for upgrades.

Elhana• 13 years ago

If Lennart Poettering were building a bicycle, it would have multiple seats, 4 wheels, engine, conditioner, bar, sat tv, console and small pool on it. You would be able to remove most of it tho... maybe. If you try to remove the engine, it would complain that modern all bicycles are broken without engines, but you can do it at your own risk.

bkor7• 13 years ago

Your comparison makes no sense at all. E.g. systemd dependencies are minimal.

My take: if he builds a bicycle it would be lean, fast and he'd analyse all existing designs to make that happen.

hkhanna2• 13 years ago

The parent comment captures why systemd makes me uneasy, and why I moved from Arch to Debian when I noticed Arch had replaced init with systemd.

The Art of Unix Programming (the book) suggests that software should do one thing and do it well, with easy, plain-text interfaces to its input and output. Systemd tries to do everything -- replacing by default so many core system components like syslog and cron.

But what really bothered me was when I realized that systemd stores all the system logfiles in binary by default! I couldn't believe it. All the system log files: boot, kern, messages, whatever, were stored in some binary format. At that point, rather than research how this new binary log file was implemented, I abandoned Arch and systemd for the init system that had served me well for 15 years.

Log files don't need to be in binary. It means we can't use our standard tools: cut, sort, grep, awk, etc. It means we can't use scripts we have written built on those tools we may have built over decades of working with UNIX.

I guess my point is, systemd, by storing things in binary format and by trying to be the 'everything' daemon, violates two core UNIX programming philosophies: use plain-text interfaces & do one thing well.

danudey• 13 years ago

Log files being in binary lets you do things like filter based on specific fields rather than on more complicated string matching. For example, you can get log file messages from postfix's smtp process (as opposed to its smtpd process) with one command (journalctl /usr/lib/postfix/smtp), which you can then cut/sort/grep/awk/etc.

You can also follow the logs with journalctl -f and go from there as well.

The problem that this solves is the classic 'what log file does this daemon log this error level to on this system', which varies a surprising amount between daemons, versions, distributions, etc.

I don't actually care which log file I look at, I just want to know what errors monit is getting. systemd and journalctl provide that handily.

I would agree with an approach to have systemd send log events to e.g. journald (or syslog-ng, or whatever the user wanted) to separate that functionality out, however.

jamesmiller5• 13 years ago

I too was put off at first when Arch moved from rc.conf to systemd as I found rc.conf simpler to use, though I trust there was a good reason to move as they are pretty good at managing the distribution.

Rachel mentioned why binary logging makes sense for complicated logs that require rather tricky state machines/regex. Apache for example, in her case.

http://rachelbythebay.com/w/2012/02/10/logs/

rodgerd5• 13 years ago

I am not a fan of binary logs, either.

I am a fan of more consistenly structured/filterable logs that are easier to pin to their service. I'm an even bigger fan of the fact systemd goes to a more Unixy model of not making every daemon author re-invent the logging wheel for their double-forking daemon.

spystath• 13 years ago

Systemd is a decent init system, fast (ssd+systemd=10 sec from button press), and dependency resolution is quite good. The units are configurable in the /etc/systemd directory and take precedence over the (/usr)/lib/systemd units. It is not perfect but it works. I believe it is somewhat better that sysvinit especially in dependencies. You just need an After=foo.service or Wants=foo.service and systemd takes care of everything. I know this is supported by sysvinit but the whole numbering thing always seemed like a hack to me. Systemd gets it right.

That being said, I can't say I am a fan of the integration of other services within systemd, it seems somewhat bloated. And I certainly don't like the whole journal thing. When the journal becomes bigger it becomes considerably slower. And I don't get the point for binary logs. I mean, logs are just text, why overcomplicate things?

I've been using it for over a year in Arch it is really good as an init daemon, I believe it is definitely a step to the right direction especially for desktops/workstations, probably for mobile as well. The rest of the package though, I am not so sure.

exDM69• 13 years ago

I expect an init system to be an init system, not to also be pm-utils, (x)inetd, acpid, syslogd, watchdog, cron and atd.

The problem with this traditional unix approach is that in addition to init, pm-utils, inetd, cron, etc you end up having thousands of lines of shell scripts that glue all this together in a big brittle mess.

While this approach may be fine for servers which boot once and stay on for ages, are connected to fast wired network and are generally rather stable configurations, I don't want such a system for my desktop, laptop or mobile device.

I expect modern devices to be able to deal with changes in power supply (battery/mains/low battery), connecting and disconnecting devices (storage, peripherals, audio devices, etc) and changes in network connectivity (intentional disconnection as well as network failure). This may involve starting and stopping services, mounting and unmounting partitions and taking other actions when the physical configuration changes.

I also expect my computers to boot and shutdown quickly and effectively.

Some of this is already doable in other systems, like the traditional Debian network setup (originally built in the era of wired networks) which can run scripts on connect/disconnect. But the situation ends up being what I described above: thousands of lines of brittle shell scripts.

I welcome the changes and the effort put in systemd and other init systems. Traditional unix init just doesn't cut it any more, in particular on personal desktop and mobile devices.

Zancarius5• 13 years ago

I had my reservations when Arch shifted to systemd. It seems like a tremendous break from Unix philosophy and a blatant violation of everything I knew.

I'm glad they did, though. Unit files are far easier to write and having a supervisor behavior in the init system alleviates a tremendous amount of administrative overhead. That's not to say shell scripts are difficult, but they can break in mysterious ways. Though I imagine the same is true for systemd, from an end user perspective, its implementation (once you understand it) requires less effort.

There are those who disagree, and I respect that. But the improvements to my desktop boot time are substantial. And writing a quick unit file to run something on boot requires little mental overhead. Just a glance over the manpage is enough to get started. sysvinit scripts on the other hand... well, maybe my memory is really poor, but if it's been more than a few months since I've written one for the specific system I'm targeting, it takes substantially longer. ;)

So, it's likely I'm stupid or systemd is easier. Or both. Probably both.

Sssnake4• 13 years ago

The problem with this traditional unix approach is that in addition to init, pm-utils, inetd, cron, etc you end up having thousands of lines of shell scripts that glue all this together in a big brittle mess.

No, I don't. Typical linux distros may, but I sure as hell don't. I have a very simple, easy to read and debug set of shell scripts that work very well: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/etc/rc?...

midas0072• 13 years ago

This is not the fault of unix, it's well suited for static systems. It's the fault of implementers trying to use something that was designed for an entirely different paradigm.

Future apps will need to copy themselves and their data to other devices, perhaps even suspend on one device and wake-up on another.

Making changes for one use case at the expense of another without knowing the history is an ignorant change for churn's sake. Churn is not productivity, but the appearance of doing something.

chousuke3• 13 years ago

The "systemd" project really isn't about having just an init system. systemd by itself is an init, but the project includes several other tools related to managing the system as a whole beyond just starting services. The project integrates a bunch of utilities that each accomplish something on their own.

To me, it makes perfect sense for these tools to be tightly integrated, both with each other and with the OS kernel, simply because they are interrelated and integration leads to greater stability and a less fragile system overall.

No doubt upstream could be less hostile towards attempts to distill out the portable bits, but I can understand their viewpoint... Systemd is ultimately about making Linux-based systems run well.

sparkie2• 13 years ago

simply because they are interrelated and integration leads to greater stability and a less fragile system overall.

This is completely counter to the way we actually design software - which is to create modular, loosely coupled components, using abstractions which allow them to adapt or be replaced, and trying to make as few assumptions as we can, since we can't predict the future. It's well understood that tight coupling is what leads to fragility and maintenance nightmares, due to decades of experience writing and maintaining such abominations.

kephra• 13 years ago

not to have everything symlinked to /usr.

an init system should be able to run !BEFORE! /usr is even mounted. Thats the division between /directory and /usr/directory, that /usr is mounted later.

Shish2k2• 13 years ago

While that's the theory, it hasn't actually worked in practice on most distros for many years -- rather than having "core stuff in root, extra stuff in /usr", we now have "core stuff in initrd, some extra stuff in root, different extra stuff in /usr, with different distros splitting the latter two up differently", the usr merge actually brings us back to the original clean design of having one "core" place and one "extras" place

midas0072• 13 years ago

And udev, which was folded into systemd.

mverwijs7• 13 years ago

I still do not understand what problem upstart or systemd are trying to solve. Slow boot times? Have you ever even booted an HP DL360? Or any server? Of the entire 8 minute or so process, sysv init only takes up 10 seconds. So what is the problem?

Containers of Debian 7 (openvz) start in seconds, with sysv init. My laptop dito.

What am I missing? Android? That takes forever to boot and I have no idea why. Could be that systemd fixes that problem.

To me, Systemd and upstart come across as modern-for-the-sake-of-being-modern.

icebraining• 13 years ago

As far as I know, it's just features, features, features: http://0pointer.de/blog/projects/why.html

agumonkey• 13 years ago

Server hardware init is also a problem, the Coreboot project started because of that (IIRC one admin mentioned something in ~hour for a cluster).

hdevalence• 13 years ago

The post linked describes quite well an example situation systemd solves, and why something like systemd is needed. Perhaps you should look at it.

otterley• 13 years ago

Isn't that an implementation detail though? sysvinit is just a series of scripts that are grouped by runlevel. It says nothing about what those scripts must do; it could very well be that a better implementation of the mounting scripts fixes the problem without having to throw out the baby with the bathwater.

dsr_2• 13 years ago

My LG G2, running android 4.2.2 with a 3.2.0 kernel, boots in about eight seconds from cold. It also has the best battery life of any smartphone I've ever seen (24+ hours, of which 8+ with screen on and doing things is normal on any day that I don't play 3D games.) SysVinit.

My boring house server, AMD FX4130, boots from a small Intel SSD in forty seconds from power to 0 load. Debian stable, sysVinit.

I understand sysVinit. I can debug it. When things go wrong, I can stop it, figure it out, fix it and start it again.

Being able to figure things out and fix them is the heart of UNIX. I'm not against progress, I'm against screwing things up in the name of progress.

OpenRC looks interesting.

atmosx2• 13 years ago

My android (Huawei G500-0100) boots in 1.5 minutes. Does that mean that we have to ditch OpenRC or whatever init system it has?

martius3• 13 years ago

You LG2 has probably optimized, vendor/hardware specific scripts, SysVinit does not achieve the same perfs on my high-end laptop.

You may want to have a look at this short conference from linux.conf.au, about systemd from the point of view of a sysadmin: http://www.youtube.com/watch?v=-97qqUHwzGM

rodgerd5• 13 years ago

I understand sysVinit. I can debug it. When things go wrong, I can stop it, figure it out, fix it and start it again.

"I don't want to learn new things. I labour under the delusion that tens of thousands of lines of shell code is a clean solution" is all I hear.

Sssnake4• 13 years ago

I still do not understand what problem upstart or systemd are trying to solve

Linux systems are still too much like unix. There are still some simple components that do a single, simple task, and can be easily debugged. This is clearly not acceptable, and does not fit with the rest of a modern linux distro. It needs to do more things, be less reliable, and be harder to figure out when something has gone wrong. Systemd and upstart both solve these problems nicely.

akira25012• 13 years ago

Well.. Lennart points this out, then walks right past it. The problem isn't the init, or the boot process, or any of that. The problem is the reliance on 'fstab'.

This is where Lennart/Freedesktop kill me; rather than notice that the underlying mechanism should be replaced, they figure they should just build a better mousetrap on top of it.

If you want to make linux more "desktopy" without screwing around too much with stupid initd decisions, then fix some of these broken underlying mechanisms.

Unless he's really suggesting that my big problem is that my /usr partition might be on a removable device.

mvaliente2001• 13 years ago

* Sh is a terrible language to build any non-trivial functionality. * System initialization has become more complex. * Currently, managing daemons has a lot boilerplate code. * That code is sub-optimal and fragile.

Personally, I don't want to know anything about the timing on which each device is available, to figure out manually the right order to start the services. That's the kind of job a program can and should do. I didn't like when I have to set IRQs manually too, or when X asked me about the data of the video chip that was connected to its bus.

Back to the init scripts. How do those scripts know if a daemon is still running? One mechanism is to check the existence of a file with the daemon's pid! IMO the current system is an anti-aesthetic patch. Only inertia keeps it alive.

This is what I would like to see in a linux system:

* A common config language for the basic services. Something like the defunct [elektra initiative](http://freedesktop.org/wiki/Software/Elektra/) * An automate mechanism to orderly start the system. * Something like Dan Berstein's daemontool to handle the daemons, plus the cron jobs. * For Ritchie's sake, keep logs as text files!

akira25012• 13 years ago

daemontools has the best philosophy for this in my opinion: just start the service. If the prerequisites are unavailable, the service will just terminate. If that happens, wait one second, then try start again.

It's perfect. No stupid 5 minute timeout like sysvinit, no stupid dependency management like systemd/upstart, very little to consider when adding new services.

If you want to use it as your init.. just look into 'runit-init'. I've been using this for 4+ years, and it's fantastic.

ahomescu1• 13 years ago

Android does not use sysv init, it uses its own init program (keeping the name, but completely different).

InclinedPlane• 13 years ago

Have you watched this video? http://www.youtube.com/watch?v=-97qqUHwzGM

It's a good introduction to the things systemd brings to the table and why it's worthwhile.

scarmig• 13 years ago

PulseAudio was wonky in 2005, therefore sysvinit forever.

sparkie2• 13 years ago

PulseAblipudio is still wonky in 2014 if you need to use it alongside JACK.

bkor7• 13 years ago

You can easily pause pulseaudio while the Jack program is running. I saw the development discussion to make that happen automatically. Further, there are various ways to do that. Seems at most a problem with how it is integrated in your distribution.

Myk267• 13 years ago

Does anyone else feel like the only reason to choose systemd is the systemd-journal? That thing is just amazing. The amount of leverage provided by not just appending to a log file somewhere will really get the juices moving in anyone who ever had to write some ad-hoc parser to do the same thing poorly.

If anyone hasn't tried out systemd or journalctl yet, I suggest running a copy of Arch Linux in a virtual machine.

Zancarius5• 13 years ago

I think the unit files are another reason. Over the years, my patience with differing ideas of how an initscript should function has waned to the point that it's almost an exercise in frustration if it's not something you regularly write on a platform you seldom use. Unit files alone make systemd a breath of fresh air, because familiarizing yourself with them requires reading only the manpages (and even then only the applicable portions). Even somewhat esoteric behaviors are readily supported without much effort.

I realize initscripts are "simple" in the sense that they borrow immediately from the shell and therefore possess all the advantages (and disadvantages) implied by such. Unfortunately, the notion of a sane initscript seems to be greatly varied in the sense that it depends largely on the distribution and the package maintainers. It isn't that they're necessarily difficult to write as much as having the power of the shell available inclines some authors to do very stupid things.

Comparatively speaking, a unit file should be exceptionally easy to understand at first blush.

chousuke3• 13 years ago

Personally, I prefer systemd. I have quite a shallow understanding of upstart, but even so I find a dependency-based model much easier to reason about than an event-based model.

I'm quite sure either would be a massive improvement over sysvinit though.

atmosx2• 13 years ago

I'm tired reading about this. It seems to boil down to politics, mostly and features less... The team with the better players will win this.

What is interesting though, is to see how the BSD ecosystem will handle this. Will they adapt in time or they will lag even further behind this?

* NetBSD might reached the point of virtual insignificance long time ago.

* OpenBSD is out of funds - which is a shame considering the software they produced (PF, OpenSSH, OpenNTPD), but their indifference for desktop makes them obsolete.

* FreeBSD (I use it in both my servers) is the most up-to-date but now with system-d/upstart not supported, I think they will run into trouble relative soon.

Darwin said it best: the ones most adaptable to change survive. Of course, the other happy scenario is that given the fact that these OSes power some big-corp servers, these big-corps will step in and fund these projects, I wouldn't hold my breath though.

rodgerd5• 13 years ago

I'm pretty sure that FreeBSD are ressurecting their OS X-derived launchd experiment to get a better init.

The argument that sysv "is Unix" is pretty much horseshit at this point. Solaris and OS X don't use a sysv style init, most of Linux doesn't any more (either upstart or systemd). At this point one is redefining "real Unix" down to NetBSD and HP-UX.

profquail• 13 years ago

Here's the link to the port-in-progress of launchd to FreeBSD, if anyone is curious:

https://github.com/rtyler/openlaunchd

Sssnake4• 13 years ago

You seem to be under the mistaken impression that netbsd and openbsd users demand the lastest in temporary, throw away, bloated, buggy shitware that linux distos are jumping on at any given moment. The fact that they don't use that crap is their main selling point. And no, netbsd is not "virtual insignifance", openbsd is not out of funds (they got over $100k just when they asked for $20k recently, and idiots take that to mean they are dead?). FreeBSD is not the most up to date, it is the most linux-like. Why do you think it is so impossible for them to also start using this pile of shit if that is what their userbase wants?

ciupicri• 13 years ago

netbsd is not "virtual insignifance"

Who uses NetBSD?

sdfjkl• 13 years ago

The sane init is one of the main selling points of the *BSDs for me. I don't want any of this make-my-desktop-linux-boot-two-seconds-faster bullshit on my servers.

What should we expect from a modern init system?

Related Stories