Structural and semantic deficiencies in the systemd architecture

http://blog.darknedgy.net/technology/2015/10/11/0/

by vezzy-fnord • 11 years ago

183 56 11 years ago

blog.darknedgy.net

dang2• 11 years ago

Since it looks like this article is trying to treat a controversial topic substantively, let's see if we can do the same. If you comment, please stick to the substance and avoid dismissive comments and rudeness. We should all do so anyway, but when the topic is polarized it can take more conscious effort.

https://news.ycombinator.com/newsguidelines.html

ncraun• 11 years ago

Very interesting article, refreshing as well to see a criticism of systemd based on technical details. Unlike most other criticisms of systemd, this author does not resort to personal attacks and insults against Lennart Poettering or fear-mongering appeals to tradition. I still think systemd is an improvement over the previous sysvinit, but the author seems to have found some real conceptual flaws with systemd that are somewhat unsettling.

I hope this article can start some actually useful conversations about init systems that do not fall into the pitfalls of FUD and negativity that seem to have surrounded this topic in the past.

striking• 11 years ago

My explanation for the lack of technical (rather than political) articles examining systemd is that it's very difficult to wrap your mind around because it carries so many ideas. A quote from an article linked within the linked article:

All of the above definitions are technically correct. None of them are complete.
It is thus that debates are doomed to go nowhere, as no one can agree on a single definition of systemd, and the developers have been largely unhelpful in that matter. Vagueness can be a strength, in that it allows for people to divide and conquer, but it is also unproductive and leads to unnecessary bloodshed.

Slowly, surely, people will begin to understand what systemd is. Only then will people be able to figure out whether they want it or not.

ausjke• 11 years ago

Read it through but hard to grasp all the details. One thing I'm sure is that for non-systemd "legacy" init systems I know what I'm doing, for systemd I most of the time don't know what's going on there, it's still like a blind art after using it for a while, overly abstracted might be one reason for that.

kbenson2• 11 years ago

I think part of the reason for this may be that some types of systems lend themselves to exploration and gradual learning through use, while others, being very tightly coupled, generally require a larger upfront investment in learning how they function as a whole, since any one subcomponent is only marginally useful without the rest.

This is largely how I remembered studying for Microsoft certification back in the day to be. There was only so much you could easily learn from playing around with the system. At some point, well written technical documentation was, if not required, extremely helpful in making connections between how systems worked together, or explaining exactly what a component was or was not capable of. I don't think this is inherently worse or better, just different, and with its own advantages and disadvantages. For example, people who went through this training seemed to have a very high base-level of understanding of how to configure these systems, what they could or could not do, and how to troubleshoot them, but not necessarily how they functioned in every case. Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked. The average person responsible for using these will have vastly different levels of skill and knowledge about them, but with that comes how it actually functions, because that's needed to troubleshoot problems, since it's so exposed.

The feeling I get mostly when reading about systemd is that it's very much in the vein of MS products. Like I said, that isn't necessarily a bad thing, but it is very different than a lot of UNIX admins are used to.

bgilroy262• 11 years ago

Conversely, the loosely coupled individual components of traditional UNIX service handlers (sysvinit, inetd) promoted people learning how the actually work, and what they do, but with scattered and not entirely comprehensive documentation on how they worked.

Can you tease out this distinction a bit more? It reads like you are saying, due to their nature, people learned how to use Unix service handlers, without a good understanding of how they were designed. It could also mean that people learned how Unix service handlers worked by using them, through trial and error, because the documentation was scattered and not entirely comprehensive.

nextos• 11 years ago

The major problem right now if one decides to use a systemd alternative is that most packages are shipping with systemd unit files, but will obviously lack support for other init implementations.

I like dmd, implemented in Scheme, which is part of the GuixSD distribution.

baghira2• 11 years ago

1. The solution to this is to write a translator from systemd units (those of .service type) to the desired kind of unit. I haven't looked into dmd, but this should be possible at least for the majority of units.

2. This was going to be a problem in any case i.e. had most distros switched to openRC one would still have been faced with the task of producing a suitable collection of service configuration files for dmd and then adapting them to dmd's specfic featureset.

mercurial2• 11 years ago

Yeah, well, if it's to go from unit files to Guile code, that's not what I'd consider an improvement. Having configuration files for use cases where the requirements are simple enough is nice.

Of course, when they're not, you end up with an abomination like MSBuild, but that's not the case here.

geofft• 11 years ago

This is a very detailed article, and I appreciate it and will need to study it carefully.

But I think that the major argument it is making is "systemd is a mediocre service manager; similar tools existed before systemd, and were better." That is a very different argument from "we would be better off with sysvinit," which seems to be the primary argument I see in practice: if you're arguing things like "do one thing and do it well," "shell scripts are great and debuggable," "init should reap children and nothing else," "there's too much code in pid1," "I don't want to learn something new," etc., you're making that form of argument.

If there is general consensus from studied opponents of systemd that something systemd-like is an obviously good idea, great. Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd. Which might even be based on an existing system like s6. But then the path forward has to be either improving systemd or replacing it, not continuing with sysvinit.

I am in practice a pro-systemd person, but that's only because my realistic alternatives, for my use case, are sysvinit and Upstart. I have no idea how I'd start to build a system that used something like s6. I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades.

If the general consensus is that a systemd-style service manager is great, but it should not be pid 1, that's an argument worth making, but I don't see much focus in this article on that.

If the general consensus is that everything can be done with sysvinit, this article doesn't make that case at all. (But I suspect that's not the general consensus.)

lobster_johnson• 11 years ago

Talking about "general concensus" is hard because there are a lot of different viewpoints, and depends on how much of the debate one is exposed to.

I have seen people compare systemd unfavourably to daemontools, s6, runit, Solaris' SMF, and Apple's launchd, to mention a few. I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative. A lot of people are unhappy with SysV, but if they don't like systemd, they would probably consider sticking with SysV to be the lesser of two evils.

Let's move on, acknowledge the network effects of systemd, and build something that can feasibly be the shinier, newer, better, securer systemd.

Unfortunately, I think that for those who don't like systemd, their criticisms (its design is a monolithic, intrusive, all-consuming, all-or-nothing ball of yarn contrary to Unix philosophy about small, composable programs cooperating over standard protocols) cannot be solved by improving systemd itself, at least not without a huge undertaking to redesign and break it apart, somehow.

rodgerd• 11 years ago

I've never seen anyone so reactionary that they suggest that SysV is in any way or shape the better alternative.

Unfortunately it's... a position that is not as rare as it ought to be. Some of the frustration and heat undoubtedly comes from the fact that while reasonable people can disagree about the best way to implement many of the goals of systemd, it's hard to take anyone seriously when they propose that SysV init is "more Unix" or technically better.

...cannot be solved by improving systemd itself

I suspect the most fundamental impediment is that upstream simply aren't interested in radical changes to how it works - it would be like trying to convince Linus that Linux needs to be a microkernel, or Theo that he needs to start using Rust for OpenBSD.

Any replacement for systemd is probably going to have to do what systemd did (and what daemontools and runit didn't achieve) and create a broad buy-in, with the additional overhead of working with at least some of the systemd dbus interfaces for compatibility.

9mit3t2m9h9a2• 11 years ago

Well, note that the article is mostly about the core systemd service management part. The article also clearly studies quality of design and implementation, accepting core goals of systemd core as they are.

One problem with «shinier, newer, better, securer systemd» is that many systemd opponents want something less shiny. And systemd is big and shiny enough that _even_ sysvinit is better.

Most arguments for sysvinit you list are, for example, arguments for s6: s6 is just a supervision suite with no other ambitions, PID 1 during the normal operation (not boot/shutdown) is s6-svscan — a 32 KiB executable, and the part that requires debugging — service launch scripts — can still be a shell script.

I am a systemd hater (at least for my patterns of desktop use — systemd can be mostly fine on some servers) and my opinion of systemd is that it is mediocre as a service management, has bad logging system bundled (is https://bugs.freedesktop.org/show_bug.cgi?id=69934#c7 fixed? can the systemd-bundled logging system give me the last 20 lines of the log without reading all 100 MiB of the log file — it previously could not?), and breaks my workflow hard (nothing similar to startx can be implemented by design http://lists.freedesktop.org/archives/systemd-devel/2012-Feb... for example), and systemd updates make systemd try to control more of the system which I explicitly do not want controlled automatically (I have a very handy binding for standby, and I do not want the notebook to suspend on lid close; I have multiple network configuration scripts for each of three typical locations and choose between them depending on whether I plan to move around more than transfer multiple-GiB files; killing all this autoconfig requires work).

I would probably use uselessd just to avoid bothering if NixOS shipped it instead of full systemd; it doesn't so it is more comfortable for me to use a NixPkgs-based system with init=/bin/sh and minimal hand-written init scripts than stock systemd. I was capable to use systemd almost fine until systemd has ensured that startx-like functionality is impossible to get.

Edit: a typo

digi_owl• 11 years ago

I suspect the startx issue is related to why they introduced the su alternative recently, PAM based session tracking.

rc4algorithm• 11 years ago

I would happily admit there is extensive room for improvement in systemd, but we UNIX folks have been working with worse-is-better and rough-consensus-and-running-code for decades.

You're understanding "worse is better" backwards. It means "starkly simple to the point of being initially off-putting is better". In this context, it means (in part) "be a big boy and learn how to write a shell script". (Not to be too condescending, but I think that captures the Unix perspective here.)

ised2• 11 years ago

Yes. I like how you use the word "initially".

hueving• 11 years ago

The article that lies herein, was, in a manner of speaking (but not literally speaking - 'in a manner of speaking' is just a phrase), so overly verbose in many places that it drowns the reader in verbal diarrhea as thick as pea soup. That is to say, I, as a reader, felt that the author (of this article) could have made these points in a more reachable, succinct fashion.

vezzy-fnord10• 11 years ago

I would, as a general principle, be cautious (in the sense of not being hasty) to dismiss in a manner reminiscent of Parkinson's law of triviality (insofar as the latter pertains to the bikeshed principle) the literary style of the article in question, since the common interpretation of excessive verbosity is flawed, in this case being seldom more than a preference with little in the way of any concrete justification that would be of direct relevance to the topic at hand, the topic being an architectural treatise. Any reservations about succinctness, or a perceived lack thereof, must therefore be qualified and left over for further analysis to draw proper and conceptually robust remarks given the evidence presented.

(Yes, the writing style is somewhat obtuse. You'll have to excuse that. The topic itself is obscure, so I suppose it fits.)

ised2• 11 years ago

"a controversial topic"

My observation: The "discussion" never seems to focus on systemd. Instead it usually turns to comments on the former sysvinit system or other init systems.

That avoidance is perhaps something to ponder. And maybe it's why this author felt the urge to put some focus on systemd itself.

Love the quote from djb. In sum, the best interface is no interface. Parsing amounts to high margin for error and often an incredible nuisance.

In another thread today I wrote about the command line interface and "expecting a reponse". Truthfully, djb's utilities that simply return an exit value and no output are the best ones I have ever used.

(It seems djb himself is a systemd user. Not sure what if anything that means.)

Imagine if the information dissiminated via www was as easy to "parse" as text0. Writing a simple "web browser" might be easy enough that programmers would not need to be paid to do it.

dang2• 11 years ago

The "discussion" never seems to focus on systemd

All the more reason to try making this one better.

(We detached this comment from https://news.ycombinator.com/item?id=10370446 and marked it off-topic.)

tadfisher• 11 years ago

My response from another discussion on this article:

Much of these deficiencies boil down to the design criteria for systemd, which is basically to avoid forcing the system administrator to perform dependency graph resolution. This was a direct response to Upstart, which pushed this work to the administrator (or distro packager) by design.

Thus the declarative syntax for unit files, which do not explicitly control dependency resolution, parallelization, or startup order, which is simply not necessary or cared about for 99% of use cases. The 1% case there is for situations where you should not use a prebaked init system in the first place, such as when you absolutely need deterministic boots, i.e. in an embedded or realtime context.

In systemd, however, the execution state is not based on explicit chain loading, but on serializing unit file options to a private ExecContext structure overlayed into Unit objects and set for service unit types.

This is not a problem with systemd's design. This is a problem with software written without the assumption of reliable process supervision, which is a reasonable assumption given the history of UNIX daemonization. systemd takes the (apparently controversial) stance that processes should not manage their own execution environment, or that of their dependencies. It provides numerous prebuilt hacks to make things work for broken software, but nobody should expect systemd to design for 100% of situations where processes do Weird Shit™.

And yes, I'm including "delegated service restarters" under this umbrella. The need for something like this absolutely smells of poor design. It also flies in the face of the author's preference for deterministic execution behavior, because by definition your process will execute differently depending on if it was started or restarted.

The link to "systemd house of horrors" [1] only proves this point.

A supervisor insisting on programmatic accommodations from service writers is not the most desirable state of affairs. A rarely discussed alternative to the two common approaches is, again as with startup speed optimization minus parallelism, checkpoint/restore: checkpoint a process image from a point where initialization is known to be complete and overlay it on startup, using a tool such as DMTCP or CRIU.

I don't know about anyone else, but this sounds like an absolute nightmare. For one, it relies on the service exposing enough information to be checkpointed at the right time; at which point, you might as well call sd_notify or hook in a script to do it for you, so this is not a reduction in requiring programmatic accommodations from service writers. Two, process-snapshotting is not a panacea, and is a nightmare to debug when it breaks your expectations; you'll need to test all sorts of process state, including open pipes/fds/sockets, signals, system calls (!), shmem buffers, etc. At which point it's best to just let the service notify you directly when the process is done. Which is a facility that systemd provides.

Of course we also have the issue of circular dependencies in the systemd architecture itself. We have the init, process manager, process supervisor, cgroup writer, local service tools, the Unit object structure (which might benefit from being made a protocol), timers, mounts, automounts and swaps all in the same module with ill-defined boundaries.

A worthy criticism. I would also like to see a bit more modularization from systemd, but I can also see the point of systemd developers not expending effort to do so. For one, it would drastically increase the number of contexts for testing (i.e. can we verify the service manager works without the cgroups module loaded, etc). I think systemd is at the point where it's rock-solid for all of its use cases, and a pluggable/composable architecture would have delayed this situation for not very much benefit.

In short, systemd imposes its opinions about process init and supervision. Whether or not this is reasonable is up for debate. However, it is very useful to have both a strong opinion and a solid implementation of this opinion.

IMO, the idealized world that systemd supports (i.e. non-forking services, initialization notification, strict execution environment and dependency declaration, centralized and opaque init/dependency-resolution/parallelization) is a vast improvement over the status quo and actually improves the lives of service writers (by removing an entire class of responsibilities), so I think it's a reasonable opinion and thus I think systemd has a reasonable design and architecture.

There are many known technical deficiencies with the implementation that are mostly being worked on, but I'm not seeing anything damning of the whole idea.

[1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...

vezzy-fnord10• 11 years ago

It appears you received a response here: https://www.reddit.com/r/linux/comments/3od733/structural_an...

You do indeed misunderstand the point about execution composition. Chain loading is indeed configuration-related, not service-related. It's a much more flexible and visible method than internal structures. Delegated restarters then can be deterministic as they reuse process management.

C/R solutions don't require special integration, that's the point. They work on unmodified processes, unlike readiness notification. This is I think a reasonable trade-off for the potential decrease in reliability, as the feature comes with a double advantage of startup optimization and then readiness notification is generally not that critical, in light of regular Unix process semantics. C/R is also constantly improving and reasonably stable for base use cases these days.

skarnet3• 11 years ago

You seem to dislike the idea of asking for notification support in the daemons themselves. Granted, it's impractical, but it's the only way to avoid race conditions which are rare for most services, but very real for a few programs that may take ages to get ready. How would you address the problem of slow starters without support from the daemons, while keeping a "the dependency graph is always respected" invariant?

colin_mccabe6• 11 years ago

tl;dr: the author argues that:

* systemd is more similar to a traditional job scheduler than a traditional init system.

* systemd should not parse text files in pid1 because it might be a security risk.

* Because systemd handles order dependencies, it is sometimes susceptible to "ordering cycles" where a clear ordering cannot be established, "dependency loops" where jobs are continuously dropped and requeued, and race conditions if dependencies are accidentally underspecified.

* "Imbalance between promoting laziness or eagerness": launchd, a predecessor of systemd, operated purely "lazily," launching services only when other services needed them. systemd also supports launching services eagerly, which the author argues is more complex.

* Checkpointing a process image could have been used instead of systemd's readiness notification mechanism (?)

* Systemd does not have a plugin system

* Journald is criticized because it does too much

I would counter that:

* I agree that systemd is similar to a job scheduler, but I don't necessarily believe that that is a bad thing. Manual scheduling of jobs for startup and shutdown like we did in the rc.sh days is complex and error-prone.

* The text files systemd is parsing are owned by root, so even if the parser is exploitable, only root can take advantage? Seems like a weak criticism.

* I agree that dependency ordering makes things more complex. But upstart and launchd have the same issues. It seems to be a tradeoff worth making.

* I don't understand the argument against supporting both laziness and eagerness in dependencies. It seems like some things are naturally modelled eagerly, like needing to perform a bunch of somewhat unrelated actions to suspend the system. And some things are better handled lazily, like setting up a FUSE filesystem when a USB stick is inserted that needs it. Shouldn't we use the model that makes the most sense?

* I don't think checkpointing a process image can ever really replace having a notification system. Even if the checkpointing code could be make bulletproof somehow, some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible.

* Regarding a plugin system: Systemd has the ability to run shell scripts, which can be useful in filling in gaps in functionality. I don't think a more complicated plugin system would be a good idea since it would add a lot of complexity (and potentially instability.)

* I didn't understand the criticism of journald, maybe someone can elaborate. The author presented some alternate approaches but I missed why these were better (other than the handwavey argument that journald was "a bottleneck")

I enjoyed reading this, and it's nice to see some more reasoned criticism of systemd. I feel like the prose got a little bit purple at times. We had to spend 10 paragraphs "descend[ing] into an inferno with the same dead horse talking points", "culminating into some rather heterodox conclusions", and "progressively introducing and elucidating on concepts, applying some a priori reasoning at times, and backtracking to derive conclusions or reiterate on prior stated knowledge" before we even saw a word of argument! This needed an editor...

vezzy-fnord10• 11 years ago

Actually the job scheduler is one aspect, complimentary to the object-oriented resource management which I made quite clear was the main point.

Regarding file ownership, not necessarily. If you run it as a session-based init, they'd be owned by the user it serves as an agent to. (see dirs: http://www.freedesktop.org/software/systemd/man/systemd.unit...)

Upstart and launchd do not have the same issues. Upstart doesn't have dependencies, but named preconditions and postconditions. launchd has neither that nor dependencies. They have many problems of their own, but different ones.

My point regarding laziness v eagerness is the way systemd implements multi-paradigm introduces undesirable externalities from both approaches, as I said: "In systemd, one must endure both the non-determinism from aggressive parallelism via dependency information, the object network needing the transaction and job metaphors, all while having its lazy features be thoroughly restricted to its internal Manager and not a flexible toolkit such as UCSPI."

I'd assume the limited use of C/R to simply cut down initialization times is feasible: http://criu.org/Usage_scenarios#Slow-boot_services_speed_up

Running shell scripts doesn't mean much when the underlying system has a large surface and ill-defined module boundaries. A plugin system most certainly does not have to be complicated: look at finit and initng for examples.

What about journald did you not understand? You lose control over the separate stages of collection, rotation and processing. Particularly having custom postprocessing is important for interoperability with foreign services that might expect certain format constraints. Having the logs be per-service instead of globally snarfed is also saner.

Yes, I did go overboard on the reservations. It's a flame war topic, so I felt it necessary as a precautionary measure.

colin_mccabe6• 11 years ago

Regarding file ownership, not necessarily. If you run it as a session-based init, they'd be owned by the user it serves as an agent to.

Interesting. I didn't realize systemd had a "user mode" where it doesn't run as pid 1. Does systemd run as root (or root equivalent) in this mode? I wasn't able to find this from a quick skim of the information available online.

Sorry, but I have to disagree. The terminology may be different, but ultimately upstart's "start on" directive establishes a dependency. Once you have a dependency you can have circular dependencies. And once you rely on the init systems to establish ordering rather than having everything just start up in a deterministic order in simple single-threaded script, you can have race conditions where things happen to work on one machine but not another. I'm not as familiar with launchd; maybe its socket activation makes the circular ordering harder to hit, but fundamentally there can always be races.

I'd assume the limited use of C/R to simply cut down initialization times is feasible: http://criu.org/Usage_scenarios#Slow-boot_services_speed_up

Perhaps for some processes this is feasible, but nothing on that site addresses my objections above: "some processes deal with state in the external world, or with hardware, that makes checkpointing infeasible." You will always need a notification mechanism even if you have checkpoint and restore. And why have the complexity of checkpointing if you have a working notification system?

From a user's point of view, I do not want a plugin system. I want systemd to work the same on SUSE vs. debian vs. Red Hat. It is usually better to put things into the base system than to have them live in a half-supported state outside the tree. That's why web browsers are killing plugins: in order to be useful, they have to reach deep into the guts of the program, which is not a place I want 3rd party code.

I don't understand why I would want separate daemons for collection and rotation, at least for dmsg, kmsg, and syslog. logrotate always had a bunch of race conditions; it was never a great design.

I agree that journald doesn't do custom processing. However, I'm a little unsure whether I would really want it to, or whether I'd be better off running my own syslogd implementation at that point. That is still an option, even when you are using the rest of systemd.

Having the logs be per-service instead of globally snarfed is also saner.

You can always set up important services to log to a file rather than to syslog.

My real criticism of systemd is that they made no effort at portability outside of Linux (and I don't buy their arguments about why it's too hard.) It would have been a lot less controversial if they had at least pointed out how the BSDs could add support (even if they themselves didn't do the work).

adekok• 11 years ago

My take-away is the following:

* systemd tries to solve a complicated problem

* systemd has a lot of complexity

* some of that complexity seems unnecessary

* parsing is hard (see DJB). Parsing (as systemd does) in PID 1 is bad. Complex systems in PID 1 is bad.

* systemd manages states and dependencies (this is good)

* some of the states are only internal, and can't be serialized to storage. The result is indeterminism and race conditions. (This is bad)

* as a result of the above point, systemd behavior is hard to debug, and hard to understand

* another result (not stated explicitly) is that systemd allows for (and apparently can't detect) ordering cycles and dependency cycles.

* The configuration files are simple for simple tasks (good), but not complex enough for complex tasks (bad)

... etc.

i.e. systemd is an engineering solution. It pretty much works, but it has inherent design / implementation issues which cause real-world problems.

Jweb_Guru• 11 years ago

This article pretty conclusively convinced me that systemd is actually good, if these are the strongest arguments against it. Most of the dependency-related stuff (process order related, cyclicstuff) seems eminently fixable. I found his arguments about dependency management having to be linearizable highly unconvincing.

sagichmal• 11 years ago

I found his arguments about dependency management having to be linearizable highly unconvincing.

What? Can you expand on this? To me it rang as a well-reasoned, and pretty devastating, critique.

9mit3t2m9h9a2• 11 years ago

These are the strongest argument against its core service management logic; those that explain why it is not better than Upstart (or daemontools or s6+s6-rc …)

And systemd causes problems not only by this, but also by its bundling of similarly mediocre or worse other components.

chris_wot• 11 years ago

Thank you for doing this - I'm going to read this with a lot of interest :-)

vezzy-fnord10• 11 years ago

Do return with commentary.

Structural and semantic deficiencies in the systemd architecture

Related Stories