Nginx 1.7.1 adds syslog support

http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log

by kolev • 12 years ago

119 43 12 years ago

nginx.org

kolev4• 12 years ago

Here's also the error_log: http://nginx.org/en/docs/ngx_core_module.html#error_log

And the changelog: http://nginx.org/en/CHANGES

staunch• 12 years ago

They had to add a checkbox to http://nginx.com/products/feature-matrix/

akurilin• 12 years ago

Why is logging to syslog as opposed to a separate file an important use case? Genuinely curious.

loadquo• 12 years ago

In short remote syslog-ing.

quonn• 12 years ago

It's useful if you want to log to a centralized logging server. This helps to have all your logs in one place and also keeps the logs safe, if someone breaks into your server.

nrr• 12 years ago

That's a pretty weak argument considering that syslog is entirely UDP and is bound to drop log data, sometimes en masse, most likely even silently. Not a good idea.

Why not use something like multilog or svlogd and wire up a tiny processor for it to kick logging data over someplace using something like rsync?

To boot, syslog is annoying to tune, depending on your particular implementation. rsyslog has a default buffer limit of 2k, whereas other syslog implementations (IIRC, syslog-ng and Solaris syslog at the very least) have default buffer limits of 1k, and this might not be obvious until you're running up against that and make the shocking discovery that you're losing data.

On an nginx server that services 2TB/mo worth of transit (which is distinctly possible since I've got infrastructure in production that does this), there's a good chance that you'll be stretching some of these limits a bit.

cones688• 12 years ago

Also allows you to correlate them with backend services, Web App Firewalls etc to see if there is an current attacks or threats.

gmjosack• 12 years ago

The simplest answer I could think of is getting remote syslogging for free. This also makes it easier to process nginx logs in realtime without named pipe trickery. I'm pretty excited to clean up some log ingestion code.

chrisbolt• 12 years ago

syslog -> logstash -> elasticsearch -> kibana

rahilsondhi2• 12 years ago

Can I reach you privately? I need help with my ELK set up :/

WatchDog• 12 years ago

I think one reason people wanted it was to reduce/eliminate disk IO.

frankzinger• 12 years ago

Yeah, and then you can also optimise the logging server's secondary storage setup for mostly appending writes. Or so I hear.

forgottenpass• 12 years ago

The use case is people with:

- A large enough deployment to want centralized logging

but are:

- Cheap enough not to buy nginx (for good reason or not)

and

- Too lazy to maintain a patcheset against distro packages

and

- Too bad at linux administration to use the file pipe trick to log to syslog anyway

So, yeah syslog is nice but this change does have quite a narrow use case. What it did have were vocal complainers that knew the right places to complain online to be noticed.

peterwwillis3• 12 years ago

This is kind of a cheap shot at people who just wanted built-in syslog support. There are plenty of reasons someone might not be able to patch or upgrade a binary, and using a fifo is a pretty bad kludge considering the whole thing can hang if they're started/stopped in the wrong order (meaning your startup scripts now have to be rewritten). Building in syslog support means just getting the damn thing working without special hacks and kludges.

But to answer OP's question, syslog is better than just appending to a file because syslog does a lot of things for you, like filtering your logs in real time and splitting them into new files, logging remotely to industry-standard aggregation devices, access control, (somewhat) standardized formatting, log rolling, etc.

worklogin• 12 years ago

My job as a sysadmin is to keep servers running and log their status and errors.

When a large team has many jobs:

-Yes, I want central logging

-I probably did buy nginx (you're telling me this was already available, but closed source? How rude.)

-No, managing distro patches and writing a web server are not my jobs

-No, I'd prefer avoiding hackish file redirects in lieu of real features, but I'll do what I have to.

mfjordvald2• 12 years ago

Especially considering that it's been available as a community patch since 0.8 days.

http://wiki.nginx.org/3rdPartyModules#Third_party_patches

kolev4• 12 years ago

True, it's been available, but having it in the core means that it will get into Linux distros, i.e. people don't have to compile.

nailer2• 12 years ago

Distros have been patching things like this for a long time.

mfjordvald2• 12 years ago

Yes, I'm saying it's really about time they added it to the core, especially because it's been available as a patch for a long time.

I get that they need to differentiate their paid version, I just wish it was in areas where the community hadn't already provided something that works.

kolev4• 12 years ago

I hope they also add cache purge into the community version as well.

nailer2• 12 years ago

Any systemd experts know if there would there be any benefit to supporting journald directly, rather than through the syslog API, on Linux hosts?

gegenschall• 12 years ago

Have a look at this[0] blogpost. Also, I'm just guessing, once kdbus is in the kernel using direct calls to journald will (or could potentially) be more efficient.

[0] http://0pointer.de/blog/projects/journal-submit.html

Andys• 12 years ago

How much buffering does it do, if any? If not sufficient, it can result in either reduced performance or dropped log messages..

peterwwillis3• 12 years ago

Why would you buffer udp packets?

ishi• 12 years ago

What does Nginx do if the log message is longer than the 1KB limit imposed by the syslog protocol?

viraptor• 12 years ago

That's an implementation defined limit in practice. rsyslog for example defaults to 2K (apparently to be compatible with upcoming RFCs), but it's configurable to higher values. If you want to be compatible with other software though, it may need to stay at 1K.

atmosx• 12 years ago

It was about time, although I won't make any use of this feature.

ps. I always found metalog so much better than syslog.

livid• 12 years ago

This is very useful when you need to consolidate error_log from hundreds of Nginx servers.

napsterbr• 12 years ago

Did someone manage to setup remote log using the buffer option? I get a "parameter "buffer=32k" is not supported by syslog" but I'm pretty sure I'm doing something wrong.

Thaxll• 12 years ago

Finally... it worked well with logstash but a native implementation is always better. Is that TCP only, I don't see any UDP settings?

peterwwillis3• 12 years ago

It's UDP only. They also hard-code the port instead of looking it up with getservbyname, which is a little weird. If you want to use TCP, send syslog messages to localhost, and have a local rsyslog/RELP forwarder send messages remotely.

stonogo• 12 years ago

getservbyname under glibc uses dlopen, effectively working around static compilation. while this is a non-issue on ubuntu, the nginx devs are aware of the number of people building against uclibc or musl to produce static binaries for embedded use. I've seen nginx running on bare metal -- just it and libc, no kernel underneath. kudos to them for making this easier than they have to.

jesalas• 12 years ago

Hooray! Whiny complaining on the Internet works sometimes!

Nginx 1.7.1 adds syslog support

Related Stories