Systemd Forward Secure Sealing of System Logs Makes Little Sense

http://louwrentius.com/systemd-forward-secure-sealing-of-system-logs-makes-little-sense.html

by louwrentius • 12 years ago

39 46 12 years ago

erhardm2• 12 years ago

Systemd is starting to show the signs of feature creep which IMHO is good. Failing early means the ship can be turned around if there's enough will, or at least (newer) replacements of systemd(in case it goes really off the rails) will learn from others' experience.

UnoriginalGuy6• 12 years ago

This was added in 2012.

erhardm2• 12 years ago

You mean the article or the feature? The article is dated by 22/11/2014. The feature may have been added a few months/years ago, but the point still stands about systemd showing signs of feature creep. Note also the DNS resolver issue[0]

[0] http://seclists.org/oss-sec/2014/q4/592

ajross• 12 years ago

Won't happen. It's turning ugly (really the ship sailed on that with the container stuff), but it's never going to "go off the rails". It's quality software and it works.

I... guess I'm sort of in the "anti-systemd" crowd, but in the mildest sense. For the overwhelming majority of use cases, it really just isn't appreciably better. It's better in lots of measurable, small, and ultimately more or less irrelevant ways. Runtime system software maintenance just wasn't a problem worth solving, IMHO.

But... it's quality software with solid features and it works. The same logic says we should just shut up and take it. So I do.

GFK_of_xmaspast• 12 years ago

I don't have any dogs in the systemd fight, but it looks like you're moving from 'anger' to 'bargaining' in Kuebler-Ross.

teddyh3• 12 years ago

This starts off looking like a reasonable criticism of systemd, or rather that systemd claims to do more that it (or any alternative) actually can guarantee. It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant).

But then it veers off: “This whole observation is not original, in the comments of the systemd author's blogpost, the same argument is made by Andrew Wyatt (two years earlier). The response from the systemd author was to block him.” No links are provided, so we can’t know what any of them actually said or did.

This smells like deliberate character assassination. Especially the fact that no references are provided seems fishy to me.

UnoriginalGuy6• 12 years ago

It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant).

I'm critical of the article also, but they in fact did. If you click the "Forward Secure Sealing (FSS)" it takes you to a G+ page with the quote on it from Lennart Poettering on Aug 20, 2012.

teddyh3• 12 years ago

It was not stated that that was the source of the quote, and the link was not given anywhere near the quote, but I guess you are right that there was a link…

So this quote is from Aug 20, 2012. Has Lennart’s opinion of this changed during the more than two years that have passed since? Does the documentation of systemd make this guarantee? Did the documentation ever make the claim that FSS is an complete replacement for remote logging?

Now that I think about it, and have the context of the quote, I think that I (and the blog post author) maybe misunderstood Lennart. FSS is not a complete replacement for remote logging, but: Among other things, remote logging used to be the only way to get immutable logs. If this was your only reason for using remote logging, then FSS is a viable replacement.

EDIT: Now that I can actually read what Andrew Wyatt wrote and how he wrote it, I can see why he was blocked. He was condescending and vague on details, and (according to Lennart) Andrew had a habit of doing that. The comments to the blog post actually cover what I wrote in my previous paragraph, so there is no excuse to dig this up now, two years later, and deliberately misunderstand it in order to gain points with the systemd haters. This is FUD.

DSMan1952762• 12 years ago

I found the comments from Andrew Wyatt - They're on the same G+ page linked in the article actually (Though it would be nice if the article pointed this out). Here's a full copy of the relevant comments:

Andrew Wyatt:

Sigh.

Lennart Poettering:

+Andrew Wyatt Sigh?

Andrew Wyatt:

+Lennart Poettering I'm not going to waste my time explaining why.

Lennart Poettering:

+Andrew Wyatt cool, then what you doing is not helpful, and just noise, and hence waste of time, and hence why do you post this at all? You are contradicting yourself there...

Andrew Wyatt:

+Lennart Poettering I forgot that this wasn't a free internet, forgive me for sighing in public about something that doesn't really solve the problem that it aims to solve.
One of the principle reasons to ship logs is not just to protect and preserve them, but to remove the control of the logs from the administrators of the systems - as sometimes the attack happens from the inside.

Lennart Poettering:

+Andrew Wyatt Cool, I'll translate this for everybody else: +Andrew Wyatt has no clue what he is talking of, but is full of negativity, hence tries to spread FUD, and be as unspecific as possible, making wild claims he makes up of thin air, to just make this negativity stick.

Lennart Poettering:

+Andrew Wyatt I never claimed that FSS tries to solve the problem of attacks from the inside. That is a fundamentally hard problem, since it is entirely against the protection model of Unix: the admin is in power, and this gives him the power to stay in power. If you want to fix that problem then an external log server won't help you at all. You need to change Unix from the ground up, and not just that as the untrusted admin problem is primarily a social problem that you can only do so much about solving technically.

Andrew Wyatt:

+Lennart Poettering it is actually a pretty easy problem to solve, I should know as someone who has solved it. Log syslogd events including starts and stops, and ship every log message to a server or cluster of servers that administrators don't control. A good forensics auditor would know who was on the server when the syslog kill event occured, and if the nodes are configured right - at least some of the steps executed prior to.

Lennart Poettering:

In case anybody wonders, I have blocked +Andrew Wyatt now. If people get personal I am more than happy to remove their comments, and ultimately block them.

UnoriginalGuy6• 12 years ago

The last reply claims they removed some comments, so we don't really know what got said between those two posts (the last two). Maybe nothing, maybe something.

angersock3• 12 years ago

So, in one fell swoop, Poettring demonstrates both his own immaturity as well as a practical example of not having an external log server--had that been the case, the comments couldnt've been deleted by a "disgruntled admin" (Poettering). :)

Why the hell would you trust a software community with this guy? He doesn't seem reasonable at all.

louwrentius3• 12 years ago

I've updated the article with a link, it's the same link as at the top of my blog post but originally it may not have been clear where I got this quote from.

UnoriginalGuy6• 12 years ago

This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user).

Also the author picks apart a quote, but if the assumptions in the previous paragraph is taken at face value (that an attack can have sup-10 second root with automated attacks on the logs), then even remote logging isn't a secure vector as they often aren't sent instantly so an attacker could intercept or alter them.

However I suggest that such a scenario is uncommon. Most attackers have to start outside and fight their way in which will generate plenty of logs for either external logging or FSS.

It seems to me that systemd now also wants to play the role as some crude intrusion detection system.

FSS isn't a "intrusion detection system." It is an attempt to make logs immutable, nothing more. The nice thing about FSS is that it can be coupled with an actual IDS and the IDS can look for attempts at altering the FSS.

If the systemd author would just have implemented sending log events to an external log server, that would have been way more useful security-wise, I think.

It isn't a binary choice. They can implement FSS and remote logging. Also it is open source so you yourself can contribute external logging.

tw042• 12 years ago

This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user).

You mean like a disgruntled system administrator? There are countless scenarios in which an attacker goes from zero access to root within 10 seconds. Predicating your security on that not being a common scenario shows that you've spent next to no time in a real corporate environment. IE: Poettering.

matthewmacleod• 12 years ago

Nonsense – defence in depth and all of that. FSS is useful because it provides a layer of audibility for users that wouldn't have external log servers in any case, and because it provides some minimal guarantees about the ability to tamper with logs. That's useful, regardless of the fact that it's not a bulletproof solution!

mikeash• 12 years ago

I don't know a whole lot about this stuff. Why would one set up remote logging and not send logs instantly?

ecnahc515• 12 years ago

Sometimes you have a lot of throughput, and it simply isn't feasible. Could potentially saturate your network, or overload your remote aggregator. Of course, all of this is fixable, but often times when every node just pushes logs out, it's hard to throttle that.

UnoriginalGuy6• 12 years ago

Because sending logs has a lot of overhead. Much of that overhead can be mitigated by waiting even a few seconds, packaging several logs together, and sending a batch.

On a LAN where the sender and receiver are close this type of overhead is much lower than over the internet. It is also worth noting that on some systems several logs can be generated each second, several hundred a minute, so even modest overhead can become costly 24/7/365.

It is also worth noting that some people have remote logging configured NOT for security reasons (primarily) but for system health monitoring. In this scenario the speed isn't as vital as keeping the overhead low.

killnine• 12 years ago

they're less common now

Source?

arent sent instantly so an attacker could intercept or alter them

intercept where?

Most attackers ... generate plenty of logs

Source?

They can implement FSS and remote logging.

Point is, thus far they have not, and their first choice was not what the author views as more secure.

Tuna-Fish2• 12 years ago

Point is, thus far they have not

This has been possibly through forwarding to rsyslog right from first release.

UnoriginalGuy6• 12 years ago

Source?

Experience. I've set up SELinux, AppArmor, etc. For example back near 2000 a lot of Apache servers I worked on were running as root, now few are, and better still many have limited permissions even as their user/group.

The same can be said for many Linux services today. And better still the few that HAVE to run as root often come shipped with some level of permission controls to limit exposure.

I can link you to a random book on Linux access controls and security if you want an actual third party source. Or be a total dick and link to a man-page.

intercept where?

It depends which system you're using to send logs remotely (and how it is configured). Sometimes the filesystem, sometimes IPC, etc.

e.g. http://www.rsyslog.com/doc/queues.html

> Most attackers have to start outside and fight their way in which will generate plenty of logs for either external logging or FSS.
Source?

Source on, what? That attackers who break into systems generate logs? You REALLY need a source on that, REALLY? Come on now.

Point is, thus far they have not, and their first choice was not what the author views as more secure.

Alright, and then criticise them for that. If the article had been about systemd's lack of remote logging that would have been one thing, but it was complaining that FSS was bad because systemd lacked remote logging. FSS existing is irrelevant.

Also, again, open source. Go implement shit.

click1703• 12 years ago

>arent sent instantly so an attacker could intercept or alter them

With Reliable Event Logging Protocol [0], interception is somewhat mitigated. You can configure and white-list certificates on the client and server so that even if a network attacker inserts a syslog relay with a certificate that's signed but isn't the one expected, you can't intercept the messages. Though, if you have that much control of the network, you can just drop syslog packets entirely. Having a dedicated management NIC and network helps mitigate that.

And I think there's a big difference between 10 seconds, and any delay inherent in to external syslogging. 10 seconds is enough time for an automated exploit to penetrate multiple layers of the security onion, but external logging should be able to squeak out a few messages between pwnage and syslog clobbering in the same situation (YMMV).

[0] http://www.rsyslog.com/doc/relp.html

meowface• 12 years ago

(Not OP, but I'll reply anyway.)

Source?

You need a reliable kernel exploit, which isn't always easy to download and compile within 15 minutes (or 10 seconds, or whatever you set the interval to). But I agree that this is not a very good mitigating factor.

intercept where?

If you have root on the box and logs are being sent to another server via syslog, you can intercept the traffic in all sorts of ways.

Source?

Read any incident response book or blog.

Point is, thus far they have not, and their first choice was not what the author views as more secure.

Yeah, I side with the blog author in this case.

steventhedev3• 12 years ago

You make a good point. FSS isn't an IDS, it's a metric to be used by an IDS.

However, that raises the question of what the hell it's doing in the init system, and not implemented by the IDS. That's a SOLID violation.

angersock3• 12 years ago

It would appear that systemd left behind SOLID engineering principles a long time ago.

louwrentius3• 12 years ago

This is EXACTLY my complaint.

teddyh3• 12 years ago

“Security” is not something you can relegate to an IDS. Security must be embedded in each program from the start.

tadfisher• 12 years ago

The answer to that is that systemd is much more than an init system.

vezzy-fnord• 12 years ago

An interesting thing to note is that systemd's FSS feature is based on a FSPRG [1] (forward-secure pseudorandom number generator) contributed by Lennart's brother Bertram Poettering, based on the latter's research paper entitled Practical Secure Logging: Seekable Sequential Key Generators [2].

[1] http://cgit.freedesktop.org/systemd/systemd/tree/src/journal...

[2] https://eprint.iacr.org/2013/397.pdf

angersock3• 12 years ago

Oh, goody, so it's homebrew crypto as well?

jimrandomh• 12 years ago

This is a neat feature, which I didn't know systemd had, and which I recall having wanted in other contexts. The criticism about the 15-minute interval is valid, but that's an implementation detail which, from the sound of it, can be fixed if someone puts in the effort. So while the author may have intended this as a pure criticism, it didn't really come off that way.

the_mitsuhiko2• 12 years ago

So the argument is: it's not good enough, so we need to throw it away?

click1703• 12 years ago

No, I think the author's point was that people shouldn't misrepresent FSS as equivalent to logging to a remote server. They can compliment each other, but they are not the same, and thinking they are can lead to serious security snafus.

louwrentius3• 12 years ago

Exactly.

KaiserPro• 12 years ago

Log signing is nice to have, but ultimately not that reliable. Systemd has quite a broad attack surface, and if the attacker has root, you're fucked anyway. It might be a useful tool for poking local sysadmins.

compromising systemd, forcing it to re-sign the logs (assuming that practicable.)

Potentially you can have an alert that checks the signature, which might be useful. However you can do something similar now, but comparing log files (local vs remote) although its not that easy or elegant.

I've yet to use systemd in anger. I'm sure when we update all 25k machines to it next year we'll become intimately familiar with all its pains.

Tuna-Fish2• 12 years ago

compromising systemd, forcing it to re-sign the logs (assuming that practicable.)

This is completely unfeasible. The point of FSS is that you sign the logs, create a new key using your old one, and then permanently throw away the old key. Once this is done, no matter what control you have on the system, you cannot re-sign the logs without the tampering being obvious.

steventhedev3• 12 years ago

I think the systemd complaints really boil down to two very real problems:

1. The authors of systemd aren't dealing with criticism well. They've allowed it to turn into a PR nightmare. Agree or disagree with them on specific issues, but they've allowed this to grow out of hand. There are nearly daily popular posts on HN, Reddit, etc. Most of these are bikeshed arguments, but if you have so many bikesheds that it becomes a daily issue on news aggregators, then you have yourself a culture problem.

2. Perhaps as a result of the above, there haven't been any limitations of the scope of systemd. This means that there is a lot of feature creep as stuff gets added in, and criticism gets swept away without considering it.

Unfortunately, I don't see an easy way to fix the culture problem. But it needs to be addressed.

EDIT: formatting

click1703• 12 years ago

Posts like yours are one of the first steps. At this point we need to address the culture problem and the people contributing to it instead of focusing on the distracting fallout.

I've tried to keep an open mind about all things systemd but after finally reading some of the exchanges people have with the systemd folks, I'm surprised systemd isn't more widely rejected as a viable alternative simply because of the attitude of some of the people currently maintaining it. You can't be an butthole to folks trying to offer valid criticisms and expect those same people take your project seriously. That just does not inspire confidence, it inspires skepticism and distrust.

I wonder if it's simply a critical mass problem, if once enough devs get peeved they will fork systemd and lead it the way they see fit sans any caustic contributions or contributors.

Cyther606• 12 years ago

It has been forked [1].

uselessd (the useless daemon, or the daemon that uses less... depending on your viewpoint) is a project to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it’s systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design

He could use better marketing, but I think removing the kitchen sink aspect of systemd and keeping the service file syntax is a step in the right direction.

[1] http://uselessd.darknedgy.net

ghshephard• 12 years ago

No security professional will be bothered to critique "FSS" because anybody worth their salt know that any log ON a server is, by definition, not trustworthy. They've known this for at least 20 years (if not longer), and everyone uses syslogd or some variant thereof.

So, in that sense, there is no controversy here, and this feature is, for the most part, completely useless.

In the comments below, Andrew Wyatt is absolutely correct and this, "Lennart Poettering" individual doesn't know what they are talking about.

xorcist• 12 years ago

What I find troublesome with journald is that they ignored everything that was done before in the log signing space. There has been some research done, even an RFC published. They implemented their own solution incompatible with everything else. Any questions about their design was met with, at best, indifference.

Since then rsyslog gained a log signing solution, sadly incompatible, but which at least to my eyes look a bit more conservative in design. We don't really have any second opinions on journald. That would be fine if it was just a hobby project, but I find it problematic to include by default in stable enterprise distributions.

Systemd Forward Secure Sealing of System Logs Makes Little Sense

Related Stories