Linux syslog on the way out?

http://www.itworld.com/it-managementstrategy/227291/linux-syslog-may-be-way-out

by jfruh • 15 years ago

26 45 15 years ago

msumpter• 15 years ago

On a compromised system you can't trust any file stored on the system. Be it text log files or binary log files. The person who compromised the system will just dump the entire binary store instead, which is normally what happens to text log files.

I use many commands (grep, tail, head, etc) to view and parse log files daily. And storing the files will just add complexity to a very simple system. Now I don't disagree that syslog needs some further advancements for access control/management. I may just be not thinking a head and what else could be possible; but I just see this interrupting my normal workflow for debugging issues.

dredmorbius7• 15 years ago

Cryptography, properly applied, allows you to verify file integrity. The real problem becomes one of key management however -- if the Journal proposal solves this issue (and it's not a simple one) then it's possible that the "has the integrity of my system logs been preserved/maintained" question can be answered.

Managing keys properly is highly non-trivial, especially on a system in which you want those keys to be available/accessible automatically at boot. Since the proposal works via signing (hashing) rather than decrypting, PKI could be used, but I still smell issues.

jerf2• 15 years ago

Cryptography allows you to verify that somebody who does not possess your private key did not modify the file. A hacker on the same system as a syslogger has the private key of the syslogger. They can "simply" (for varying values of simply) replay the logs again, only with whatever events they want filtered out or modified, and it will pass the crypto checks just fine. You have to offload your logs somewhere else, at which point the whole encryption in the logs themselves are sort of redundant.

Managing keys in this case isn't just "highly non-trivial", it's flat out impossible.

tincholio• 15 years ago

Do these guys even have an editor? From the article:

...nature of the syslog, which basically excepts text strings in whatever form the application...

...one daemon may send information about an even in one way, and another daemon in a completely different way...

spc476• 15 years ago

There is no method currently defined to forward logs to another machine, so the journal is definitely prone to a local attack wiping it out (you typically have to be root to wipe log files anyway). The only method mentioned to move logs off-system is scp or rsync or some other method of copying a file over the network, which, to me, is a non-starter. I keep logs both locally and forward them to a trusted host, just in case.

I'm neutral on the binary format of the log; I can see both sides of the argument.

dredmorbius7• 15 years ago

My first (shallow) read of the proposal is that it solves one relatively high-risk, low-probability case by throwing a great deal of complexity at standard operations on which a great deal of existing infrastructure exists: log analyzers, summarizers, rotation systems (broken as they may be), and just simple shell tools.

Immediately prior to launching myself into Linux, I briefly toyed with Windows NT 4.0 WS, and among the bigger disappointments of it was the system logging facility and its user interfaces. While *nix logging has its bugaboos, the ability to chain simple tools together to perform tasks (example: grep, cut, and uniq -c on a postfix logfile to give a per-hour summary of mail delivery status by various classifications) provides tremendous flexibility. As compared with the single GUI interface Microsoft provided. I'd hope that Poettering et al won't repeat this bone-headed mistake, but the simple observation that their sample log event lacks ... a date stamp??!!! ... doesn't inspire me.

If there's a useful element from this it's the list of horribles against current syslog weaknesses. Many of these could be addressed by amending syslog and/or providing a bolt-on solution. I don't see a need for a total replacement.

As to the observation that there are currently several (incompatible) logging systems on Linux systems, ob xkcd: http://xkcd.com/927/

rbanffy3• 15 years ago

A second read makes me believe it solves no problem except disk space usage. The fact it's binary and signed protects against nothing - whoever has control of the machine also controls the signing keys of the log and can generate whatever log they want, with whatever event they want.

Are we doomed to repeat every stupid mistake Microsoft made? Is the "if you don't know Windows you are doomed to reimplement its bugs" the new thing?

dredmorbius7• 15 years ago

The solution to integrity, IMO, is distributing the log (multiple independent logservers). You can hash individual line items if you want, or sign them, or do any number of other things to make it easier to find discrepancies. There's the problem of log lossieness in general, which is a tough nut to crack. Some form of log entry UUID might also be useful (though hostname + millisecond/jiffy timestamp comes close).

And count me in among those who find the bug-for-bug reimplementation of Microsoft's fatal errors just a tad annoying.

maratd• 15 years ago

This is a waste of time. Syslog is the standard for logging on *nix. You will bork everything if you replace it with something incompatible. Everybody knows this. So whatever you replace it with must be Syslog compatible. If that's the case ... there are numerous alternatives to choose from. Like rsyslog. Which is already standard on RHEL, nevermind Fedora. Rsyslog has this nifty feature where you can immediately transfer all logs via TCP or UDP to another host. Hacking problem solved. Stop fixing things that aren't broken.

naner2• 15 years ago

I disagree. A lot of these old *nix tools are showing their age and could use a refresh based on known deficiencies. Creating new tools won't harm the old ones so I don't see a problem.

rbanffy3• 15 years ago

> Creating new tools won't harm the old ones

Except when they become the default tool despite known deficiencies the previous ones didn't have.

rhizome3• 15 years ago

What are a few examples of other "old *nix tools" that are showing their age and have "known deficiencies?"

jbk• 15 years ago

Not to mention that Journal will depend on systemd which is Linux-only :D

rhizome3• 15 years ago

Over the summer I worked at a company with so little institutional knowledge that an argument broke out about whether to use NTP or not, and what to replace it with (in a webapp, natch) to ensure a good timebase. I was so flustered by this line of discussion that I had trouble participating.

InclinedPlane2• 15 years ago

What exactly was the argument against NTP?

lanstein• 15 years ago

If this means that syslog-ng is in the way in, great!

siculars• 15 years ago

Just because you make something "binary" does not mean it is inherently secure. As pointed out elsewhere, syslogd is a pretty reliable, simple and flexible foundational piece of infrastructure. Changing this with something un-tested and not entirely understood would, imho, be a Very Bad Thing(TM).

saljam• 15 years ago

Plan 9 has an extra permission for files; "append." When set, this means programs are only allowed to append text to the end of the file. Both simplifying logging (just open the file and write) and making it more secure (no arbitrary writes.)

I wonder if Lennart and Kay would accept a solution that simple...

bingaling• 15 years ago

That's the curious thing about syslogging to a network log server, it is append only (assuming the log server isn't compromised.)

It is an existing solution that simple.

wmf2• 15 years ago

Linux also has append-only files, although as usual root can override it.

ciupicri• 15 years ago

Are you referring to chatattr +a log? The problem with these files is that they can't be renamed:

    # mv log log.old
    mv: cannot move `log' to `log.old': Operation not permitted

pstoneman• 15 years ago

The bit I really don't like is that it enforces systemd - and it sounds like it's part of it. I think one of the strengths of Unix/Linux is the architecture of many simple tools, each doing one job well, and one job only - which this sounds like it violates? Ick.

jleader• 15 years ago

It makes me smile sadly when one of the problems cited is "Syslog is only one of many logging systems on a Linux machine", and the solution proposed is the addition of another logging system.

wmf2• 15 years ago

I think they're proposing to remove all the others.

Karunamon• 15 years ago

http://xkcd.com/927/

Ziomislaw• 15 years ago

Is it just me? Or do They want to make un*x more windoze-like?

viraptor2• 15 years ago

Depends on what do you mean by that. Eventlog has both good and bad parts. If they integrate only the good ones, I call it improvement, not "more windoze-like".

plq• 15 years ago

Syslog is definitely not on its way out. It's been in use for 30 years because it works. Perhaps this is rather Redhat trying its hand at the embrace & extend game by introducing proprietary tools? Forcing staff to train on such technologies (with pretty yet dysfunctional guis) will make migrating to other systems harder.

As for the free-form text "problem" this solution claims to address: Different daemons produce logs in different format simply because one's data won't fit in another's schema. And you can actually live with postfix and qmail producing logs in different formats about the same event because you have strictly one of them installed in your server. And in my experience, switching from one to another is a huge undertaking where rewriting your regexp hacks is a drop in a bucket.

viraptor2• 15 years ago

I'd love to see something like that implemented - for the benefit of having semi-structured messages... Oh wait, there are lots of daemons which already do that. And they use properly documented formats, like GELF. And they support forwarding over network. And they store data inside a proper database, instead of yet-another-file-format.

I'd much rather they simply blessed one of the existing solutions (greylog2, fluent, loads of others) and created a compatible minimal daemon which can be included by default in a distribution. Currently they're neither providing new good solution, nor solving the structured log problem (flat k/v? why not just go json/bson?). They're also trying to solve single-host security, which seems pretty close to impossible without moving logging into something like TPM - just send the logs outside, so that the logging channel is append only.

There is also one point which makes me really worried about what the hell are they thinking... "The timestamps generally do not carry timezone information, even though some newer specifications define support for it." Simple solution is to log in UTC, always, everywhere. Run your servers in UTC too - unless it's your local desktop, there's no reason not to.

inopinatus• 15 years ago

Mm. So many nails in this coffin.

Many times I've had a need to grep or tail a log in a hurry. This kind of change compromises the entire modus operandi of good Unix admins.

There must be tens of thousands of Unix-based tools out there that use the syslog(3) interface and I'm damn sure that most of them will not change Just Because Someone At Fedora wants them to.

I don't believe a binary format is any more inherently secure than a textual one. Want non-repudiation of logs? Sign them and send them off-box.

They would be much smarter to introduce incremental improvements i.e. extend the existing syslog protocol to accept a structured data format; introduce a log signing journal for integrity checks.

pjscott• 15 years ago

They're going to need really good import and export tools. If they can provide those, then most of the objections to the change go away. If they can't, then I doubt it will gain traction.

jcmhn• 15 years ago

This is the same sort of thinking that is trying to add a registry to linux as a replacement for /etc

Linux syslog on the way out?

Related Stories