“I wish systemd would get over its thing about syslog”

http://utcc.utoronto.ca/~cks/space/blog/linux/SystemdAndSyslog

by zdw • 11 years ago

95 54 11 years ago

utcc.utoronto.ca

liveoneggs• 11 years ago

You can sub syslog with any component systemd has cannibalized and write the exact same article.

That's what the whole argument was all about.

feld2• 11 years ago

The people who say "it's not that bad" are those who haven't interacted with these extremely reliable and integral parts of a Unix system for years and years and understand the ramifications of bad behavior.

_delirium• 11 years ago

The author of this article has done so, and seems to think that until this syslog interaction it hasn't been too bad:

One of systemd's strengths until now has been that it played relatively well (sometimes extremely well) with existing systems, warts and all.

feld2• 11 years ago

So this means any logging/auditing software that requires the data to be sourced from syslog is unreliable. This kind of behavior would make an auditor's head spin.

richardwhiuk• 11 years ago

If you are auditing systems based on systemd (or sysvinit or makefiles or upstart or launchd) then you are probably doing it wrong anyway

unethical_ban• 11 years ago

If you build a custom logger into a startup/scheduler instead of using the existing logging component, you're doing it wrong.

Really, if you're not using the logging mechanism to check a system, what are you using?

icebraining• 11 years ago

So systemd has a few bugs that affects its interaction with syslog. I see nothing to indicate this is specifically because of some ill will towards it.

Is there a bug report with a dismissive reply? Or one being ignored?

(I do agree that it shows that systemd is too immature for critical production environments)

JoshTriplett• 11 years ago

Is there a bug report with a dismissive reply? Or one being ignored?

That's always my reaction to blog posts like this as well. A blog post is not a bug report. Given the mentioned bug in the systemd/syslog interaction that made it into RHEL, there should be a bug in the RHEL bug-tracker, and I'd hope it'd be a high priority one.

cpncrunch2• 11 years ago

I think the issue is just stability in general. I just moved to a new server recently, and I decided to go with CentOS 6 instead of 7, as people were reporting that CentOS 7 is just a little less stable in general than 6. (Although systemd probably isn't entirely responsible for the lower stability in CentOS 7, it probably is one of the major changes in CentOS 7 so it likely takes a good share of the blame).

You always end up with bugs in released software, even CentOS. Some bugs only show up in certain situations, and might be difficult to track down. The only way to give a good guarantee of stability is for a large number of people to use the damn thing for years on end!

onli• 11 years ago

If systemd had no ill will towards syslog, there would be no journald.

boklm2• 11 years ago

No, journald has its own benefits. Creating an alternative to some software doesn't mean you have ill will towards it.

erikb• 11 years ago

I only watched part of the systemd wars from the side line and to this day I haven't come up with a reasonable solution. Both sides have reasonable arguments.

Saying that a tool is unsupportive towards another tool because in the interaction two irregular bugs happen sounds like a weak claim though. Don't all tools have bugs? What makes these two special?

digi_owl4• 11 years ago

And the PR division is already in action in the comments...

pavanky• 11 years ago

The PR division is the entire userbase of Fedora, Arch and a few other distributions who have been using it for years without any issues.

pessimizer• 11 years ago

No, it's not. Just because you use Fedora or Arch doesn't imply that you aggressively support every architectural choice that they have made.

deelowe• 11 years ago

I use arch and still despise it. The reduced boot time isn't worth the headache the overly complex systemd suit of tools has introduced. BSD might have some issues, but it worked very well for my use-cases.

vezzy-fnord3• 11 years ago

I think journald is by far the buggiest component systemd has. Log corruptions, the dubious Forward Secure Sealing mechanism, journal file fragmentation, the gateway daemon, coredump hooks. There's been several CVEs for it, too. Of course, you cannot disable it without source code modification.

Quoting from the systemd wiki [1]:

   Note that it is now the journal that listens on /dev/log,
   no longer the BSD syslog daemon directly. If your logging
   daemon wants to get access to all logging data then it
   should listen on /run/systemd/journal/log instead via
   the syslog.socket unit file that is shipped along with
   systemd. On a systemd system it is no longer OK to listen
   on /dev/log directly, and your daemon may not bind to the
   /run/systemd/journal/syslog socket on its own. If you do
   that then you will lose logging from STDOUT/STDERR of
   services (as well as other stuff).

   ...unless your syslog implementation is socket
   activatable many services will not be able to log to your
   syslog implementation and early boot messages are lost
   entirely to your syslog implementation.

So the fact that keeping your regular syslogd wasn't as simple as merely symlinking syslog.service was publicly known, but often overlooked by proponents during discussions.

[1] https://wiki.freedesktop.org/www/Software/systemd/syslog/

Qantourisc• 11 years ago

That's just bad, this is breaking API's ... suddenly other logging deamons need to change their code because systemd is squatting the log file ?!?

smcleod• 11 years ago

SystemD makes me feel weird.

When I boot Debian 8 (which has systemd) I'm amazed how quickly it boots. Systemd units are nice. SysVinit was a mess.

I wish I could interrupt stuck processes when booting / shutting down: http://i.imgur.com/a4ImjAk.jpg

When someone comments about how overly complex and controlling systemd is for an init system, I think of all the hideous shell scripts that don't talk to their neighbors which end up forming most the sysvinit boot process on many distros (in the past?).

I wish there was something [lighter|simpler|more portable|more unix] than systemd.

Every time I seed something [good|bad] about systemd it makes me want to [comment|upvote|walk away].

digi_owl4• 11 years ago

There have been several over the years, but none of them had RH riding shotgun.

vezzy-fnord3• 11 years ago

I wish there was something [lighter|simpler|more portable|more unix] than systemd.

Your best bets are probably s6 or nosh. If you want just a service framework without init, then you can look at OpenRC, too.

http://skarnet.org/software/s6/

http://homepage.ntlworld.com/jonathan.deboynepollard/Softwar...

mrmondo• 11 years ago

Thanks for the links - I'm sticking with Debian for now so I guess I'm using systemd for the foreseeable future there, which I [do|dont] have a problem with. systemd hasn't failed me yet - it really does just make me feel 'weird'.

joshuapants• 11 years ago

There's always http://smarden.org/runit/

cturner2• 11 years ago

SysVinit was a mess.

Not just SysV - every attempt to move beyond BSD-style init has been a mess.

I wish there was something [lighter|simpler|more portable|more unix] than systemd.

I think the trick with systemd is to realise that it's not unix. But it's something else that's really interesting in its own right.

Init/scheduling is a much more difficult problem than it first appears, and it has always and everywhere been tacked on to its host OS as an afterthought: init scripts, cron, windows services, system v init. All are poor, partial solutions. Yuck.

To fill the void, there has been a proliferation of application containers, each of which does a different 70% of the general problem. All of this stuff could just be done in the platform. And that's what systemd is.

There's going to be two separate camps at the end of this: a group led by the BSD platforms will stick to convenitional unix goals. On the other side, the linux kernel plays second-fiddle to systemd. The systemd service graph is a new operating system tradition.

[New, but not without precedent: in the debian world, the 'apt' package management system has become more significant than the kernel. You can create things that are not linux, but which are still debian. Both systemd and apt use a sophisticated data-graph to contain a difficult problem.]

There are exciting opportunities for developers on a systemd operating system: you'll be able to quickly put together application stacks using microservices in a way that hasn't been practical until now. It /is/ unixy in the sense that it gives the console operator a powerful new general-purpose tool.

vezzy-fnord3• 11 years ago

I don't follow.

Application containers were born out of a variety of reasons, but primarily seem to be oriented around partitioning a host OS into individual resource namespaces each with its own library context. Thus, an attempt to address the fact that the host OS is deficient in providing a multi-tenant experience out of the box.

In fact, application container deployments frequently don't use an init and process supervisor at all, but when they do it's a bare minimum loop that reaps zombies and reexecs a process on SIGCHLD.

On the linux side, the kernel will start to play second fiddle to the systemd.

That much is true, for better or for worse. systemd and much of the new desktop userland is refactoring a lot of kernel components into userspace, creating the de facto equivalent of an unwieldy hybrid kernel. I suppose Tanenbaum did win that debate, after all. Sort of.

in the debian world, the package management system has become more significant than the kernel.

The package management infrastructure is basically what makes a Linux distribution a "distribution". It's the backbone. That you can create Debian stacks based on non-Linux kernels is because many Linux packages are cross-Unix.

as a development platform you'll be able to quickly put together application stacks using microserves in a way that hasn't been practical until now

Wikipedia defines microservices as:

   In computing, microservices is a software architecture
   style, in which complex applications are composed of
   small, independent processes communicating with each
   other using language-agnostic APIs. These services are
   small, highly decoupled and focus on doing a small task.

That sounds a lot like... the Unix philosophy, which you openly admit systemd is in opposition to.

It seems like a daemontools-style toolkit approach (e.g. s6, nosh, perp, runit, etc.) fits the "microservices" umbrella far better. Indeed, I've been seeing plenty of container configurations based on those tools.

ChuckMcM• 11 years ago

I think the trick with systemd is to realise that it's not unix.

I keep coming back to that. It seems to me that there is a tension in the Linux world about people who want an open source UNIX and people who want an open source Windows. Systemd is much more a "Windows" approach to system configuration management than a "UNIX" approach.

That isn't "bad" per se but it makes for some confusing things. I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS. I believe such an explicit step would help folks move forward with "where" a certain idiom of OS management should land in the open source world. Of course I don't expect that to happen :-).

hobarrera3• 11 years ago

I like the rationale that services should be represented by a graph, and I'd like to see this pick up on BSD (it's not terribly complicated to implement on top of init).

However, systemd goes way beyond that, and tries to control everything too much, in too granular detail. I wish the move from init -> systemd had been more gradual, there was no need to rewrite absolutely everything around init.

cpncrunch2• 11 years ago

For the servers I run, boot time isn't an issue. I might reboot a server once a year, and booting in 10 seconds vs 40 seconds isn't really an issue. What is an issue for me is stability. I don't really care about "hideous shell scripts" as long as they work.

hobarrera3• 11 years ago

But linux isn't all about servers, there's a lot of desktop users out there.

And having desktop and server diverge at a kernel/init level seems too much burden to maintain.

rwmj2• 11 years ago

Virtual machines, containers and appliances all require very short boot times, because they may be created and booted frequently.

RexRollman• 11 years ago

The whole blog is interesting to read. Thanks OP.

LinuXY• 11 years ago

While I agree that systemd does not play nicely with syslog, you quickly realize that journald can be exported as JSON, filtered, described as time ranges, chunked and easily imported into ELK. My users (mainly developers) with less sed/awk fu have come to rely on these features heavily. I'm not really sure what there is to miss about syslog or syslog-ng when you have a robust ELK+journald system.

rwmj2• 11 years ago

Unfortunately it's also very slow. Text files don't have the many advantages you outline, but in everyday use they are much faster than the C journal API.

“I wish systemd would get over its thing about syslog”

Related Stories