Guide to Logging

https://www.loggly.com/ultimate-guide/

by makaimc • 11 years ago

205 92 11 years ago

acheron• 11 years ago

My four-year-old watches this show, whence I've learned all about logging: http://www.amazon.com/Mighty-Machines-In-The-Forest/dp/B0000...

0xdeadbeefbabe• 11 years ago

The subtext that I've just started to notice, as I watch with my four year old, is that Canada is awesome.

DKnoll2• 11 years ago

I'm going to take this even further off-topic but if you enjoy cheesy vintage Canadian children's television Hammy Hamster AKA Tales of the Riverbank is required viewing.

mugsie5• 11 years ago

https://www.loggly.com/ultimate-guide/?s=parse+WSGI+logs&pos...

0 results

https://www.loggly.com/ultimate-guide/?s=parse+python+logs&p...

Gives you a page on NodeJS

Ultimate ... not so much.

mentat• 11 years ago

For even "haproxy". Surely no one uses that right?

mugsie5• 11 years ago

did a quick scan - nginx just returns generic "linux" / "systemd" (and node for some reason).

ruby - you guessed it - node.

jt2190• 11 years ago

I've downvoted you because the world doesn't need yet another post about how some open source project that has the audacity to call themselves "the ultimate resource" for their initial release doesn't live up to some-random-person-on-the-internet's idea of "ultimate".

I'm guessing from the specifics of your example queries that you already have some knowledge about WSGI and Python that you could contribute to the guide. Why not do that instead?

mostlyjason7• 11 years ago

Great suggestion! I'm one of the tech editors for the guide and we'd love to have contributors for python and other languages! http://www.loggly.com/ultimate-guide/become-a-contributor/

mugsie5• 11 years ago

And I commented, as it calls itself ultimate, and it has articles about Java, Node, Apache, Linux (including SystemD) and Windows.

If it was titled "We want to build the Ultimate Logging Resource" I would be OK with it.

I would have no problem contributing, and may well do so over the next while, but titles like this get my goat

Edit: Also, while I know this will be downvoted to hell - Open Source? It is by a commercial company, with -all- most of the examples being their product.

jedberg24• 11 years ago

I have a long rant about logging, but the TL;DR is: aggregated logging in general is useless don't bother, everything you need to know should be in your monitoring system. Basically, if something is broken and it's a problem, it will happen again, so you just start logging when something breaks. How do you know something breaks? Log your application metrics. Log your customer experience in the form of metrics and monitoring.

But don't aggregate every code exception and warning. If they aren't affecting your customer experience, who cares? And if they are affecting it, then it should be expressed as an application metric in your monitoring system.

Edit: To make my point a little clearer, I believe that central logging and aggregating should be available, just not turned on all the time.

teacup50• 11 years ago

What?

The only way to find many issues is by looking at logs of unexpected (and especially unhandled) exceptions.

If you have so many of those that they're impossible to dig through, FIX YOUR CODE.

jedberg24• 11 years ago

You can keep the logs on the local box and take a peek at them once in a while if you want to clean up the code (which you should), but there is no reason to spend a ton of money and time aggregating the logs.

EvanAnderson• 11 years ago

This perspective completely ignores the use case of off-the-shelf software where you have limited or no capability to add "application metrics". Logs are invaluable in that scenario, which is arguably the majority of software deployments.

jedberg24• 11 years ago

You can extract application metrics from the local logs as they are output (and then redirected to /dev/null).

jacques_chester4• 11 years ago

Logging is for events with identity, metrics are for events without identity. They are not the same thing.

Sometimes I care about the bulk behaviour of my application. I hit 2757 RPS and suddenly find a catastrophic gap in the performance landscape of my application -- that's useful. I don't care much about the particulars of the requests. What matters is the statistical behaviour.

But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details.

Metrics are warnings that smokers die younger, on average, than non-smokers.

Logs are a biopsy showing which kind of lung cancer you have.

No sensible production system or PaaS treats them as identical, even if you can derive metrics from logs.

jedberg24• 11 years ago

But sometimes the application crashes when a particular input is given. These are outliers, they don't show up clearly in statistical behaviour. In that case I care about the details.

But my question is, why do you care about the outliers if the customer experience is not harmed by them? And if it is harmed, then it should should up in your 99th percentile metrics, even as an outlier, at which point you can turn on aggregated logging to look for that one thing. And if that one thing doesn't happen again, then did it really matter?

henrik_w2• 11 years ago

Well put! Also, there are the cases when the system does not behave as expected (for example, mis-configuration, or wrong input from the user) - in those cases there are no exceptions or crashes, but a good log will tell you why it happens.

I've given a more detailed example of this in example 2 here: http://henrikwarne.com/2013/05/05/great-programmers-write-de...

bbrazil• 11 years ago

An additional practical difference is that due to tracking less, that metrics can take much fewer resources allowing you to gain insight that's not practical from logs.

I'd expect in a well instrumented system that a single request could easily affect hundreds of metrics. Trying to get that level of information from logging alone isn't practical due to the volume of data it'd require.

If you want to track overall behaviour of hundreds of codepoints then metrics are good, if you want to track behaviour for all of your events at a handful of codepoints then logs are good.

kiyoto3• 11 years ago

I think you are conflating between the needs of centralizing logs and having them around at all. The OP is saying that always centralizing them might not always make sense, and I tent to agree (and I say this as someone who maintains a popular open source log collector)

If I were to play the devil's advocate, the real needs for raw log data in a centralized location is for folks outside of Ops: data analysts and data scientists.

mugsie5• 11 years ago

Couple of reasons - 1. You need to stick to compliance requirements. 2. Tracking individual transactions (user complains about $Problem - if you attach a request ID to each request, and pass it between all components, you can find the exact problem for that request) 3. Building retrospective graphs is much easier when you have the data to hand, which you can them opitimise into a metric.

Logging is not the only solution, but neither is metrics / monitoring only.

henrik_w2• 11 years ago

Where I used to work, we had a great system for logging/tracing only sessions we were interested in, while not generating any output for all the other sessions. I wrote about it here: http://henrikwarne.com/2014/01/21/session-based-logging/

kiyoto3• 11 years ago

I think there are two kinds of logging that's conflated into two in the industry: logging for devops and logging for analytics.

For logging for devops, I 100% agree with you. Looking at application metrics rather than raw logs is far more productive, and the raw logs should only be consulted after you have triaged the situation based on the metrics monitored.

However, there is another kind of logging, and that's for data science and analytics. Here, it's hugely helpful to have centralized logging. Hell, it is a must. The last thing you want is to have data scientists with a shaky Linux knowledge to ssh into your prod machines. At the same time, logs are the best source of customer behavior data to inform product insights, etc. By centralizing these logs and making them available on S3 or HDFS or something, you can point them there and have everyone win.

Among Fluentd users, we definitely see both camps. As a matter of fact, one of the reasons that I think people like Fluentd is that because it enables both monitoring and log aggregation within it.

a3voices2• 11 years ago

One alternative is to replicate some of your data into a different DB that is safe for the data scientists to use. And that way, they have raw data to play with instead of logs.

jedberg24• 11 years ago

You're absolutely right. I was only talking about DevOps logging (should have made that clearer). Logging for data science is a totally different ball game.

ketralnis• 11 years ago

Unless you think you can predict every single bug that you'll ever write, and also how it will affect your customers' experience, how do you know you can track it at all?

And if you're not logging it, once you do figure out that 50% of your customers are leaving because your payment endpoint throws an exception on Amex cards, how do you know what's causing that? Now it's just a silly detective game for something you could have been just logging in the first place.

And if you're only able to find out via your customer metrics, it's already too late. If you were just logging the exception, you'd have said "huh that's funny" when you deployed the code, rather than when you have enough data to realise that you've already lost the customers.

Personally I'd rather just have the logs.

jedberg24• 11 years ago

If you were just logging the exception, you'd have said "huh that's funny" when you deployed the code, rather than when you have enough data to realise that you've already lost the customers.

I'm assuming that you still have the local logs on the local server and you're looking at one instance as you deploy to find those exceptions.

I'm also assuming that we're talking about a system too complex to actually watch logs go by in real time.

If the log level is light enough that you can actually read the logs as you deploy, then by all means do so. But if you're aggregating them just to be able to search them because you can't read them as they go by, then I'm saying there are better things to be doing than storing all those logs.

pbowyer2• 11 years ago

Having just started to go down the 'log everything centrally' route (to keep track of when third party services are down, what actions User 1234 performed the day they reported an issue, and which transactional emails seem to have disappeared), can you explain how I'd change this from logging to metrics & monitoring?

One driver has been the lack of information I've had for debugging production issues (e.g. user X not able to log in against authentication provider Y) and so storing extra info is needed (vs the standard web server logs)

pbowyer2• 11 years ago

Thinking further, I guess I'm conflating logging for devops and logging for analytics and logging for audit trails.

Not sure how to separate it out without data duplication (the boundaries aren't clear cut?) and for a small system it prob doesn't matter - I'm nowhere near Netflix, PayPal et al for scale.

mirimir• 11 years ago

I find nothing on tor or lighttpd :(

avn2109• 11 years ago

Was I the only one who was really hoping this was going to be about best practices for cutting down trees?

mostlyjason7• 11 years ago

Hey guys I'm one of the technical editors for the guide. We created it because logging is really valuable for troubleshooting and monitoring systems. However, it's a complex topic and so many resources today are spread around on different forums. We wanted to have a place to consolidate all that useful information into something much easier to read. If it helps a dev or ops person solve a problem faster or easier than we've done our job!

There is still a lot of content to be written for languages like python, ruby, and more. We'd love to have more contributors, or even just suggestions for the site. I can also get in touch with any of the authors if you have questions for them.

kordless2• 11 years ago

many resources today are spread around on different forums

Centralizing the all the knowledge that's been written about a given topic isn't scalable in terms of contributions. It's far better to focus on enabling contributions than it is to focus on building a solution in which to put the contributions. There's a reason products like StackExchange are more successful at providing features for encouraging contributions than topic specific sites.

There is still a lot of content to be written for languages like python, ruby, and more.. We'd love to have more contributors,

There is already a ton of contributions to logging solutions on the Internet. Why not build a solution categorizing those, instead of trying to build new content in a centralized (and I might add commercially curated) solution?

Also, where's the source to this "open source" project? I'm not seeing any links to code.

mostlyjason7• 11 years ago

I think readability is a concern. Often when I'm learning a new technology I'd rather sit down with a good book on the topic rather than reading a bunch of forum posts. I can be pretty confident that it will cover the most important concepts, building up from basics to more advanced ones, and showing all the connections in between. A lot of times reading regular posts, you have to build all these connections yourself. We do link to really good resources and documentation where they are available though so if you have any to suggest please let me know!

I think the open source part of the title refers to the fact that we cover all different technologies from a variety of perspectives, not just a single vendor solution. Also, we allow contributions from anyone in the community. I couldn't find an easy way for WordPress to share or collaborate on source code, but if you think of a way, let me know. In the meantime, we're giving access to people who want to contribute.

strickjb9• 11 years ago

Your site's readability would be greatly improved if you darkened the text color. #444 or darker is good enough for a white background.

DKnoll2• 11 years ago

Loggly is crazy expensive for what they have on offer (at least given my use case which is aggregating syslog and Windows event logs). You're much better setting up Logstash or Graylog2 on-premises or in public cloud.

packetized• 11 years ago

'Become a Contributor' seems to mean 'please give us your email address to be added to our mailing list.'

This seems moderately suspect.

Guide to Logging

Related Stories