Exception Monitoring and Response

http://githubengineering.com/exception-monitoring-and-response/

by samlambert • 11 years ago

109 20 11 years ago

spydum• 11 years ago

well, disappoint.. show a shiny tool, then say how it took 6+ years to develop, and wont likely be open sourced. yes.. lot of us would like to implement something similar, just dont have the engineering resources to flush it out.

if I had to take a stab at it, I'd say a good place to start is some ELK platform. Seems like it has almost all of the right components to build something like this. Just curious if they have their apps directly feeding exceptions, or if they are slurping logs.

aroben• 11 years ago

We send exceptions to Haystack directly. It usually happens in a begin/rescue block or via a middleware that reports uncaught exceptions.

zeeg3• 11 years ago

It's very hard to do this kind of thing by slurping logs (and is generally why ELK isn't considered a replacement for these tools). The first thing applications like Haystack try to solve is context. While I can't speak for Haystack (as I've never used it), Sentry using it's SDKs pulls in a ton of information that you would never get, or would at least be very hard to pull from logs. Examples of this include most of the HTTP request information, the authenticated user, context locals and surrounding code in the stacktrace, as well as any arbitrary context you might want to associate with it.

brianr• 11 years ago

Great to see some more brainpower going into this problem. We at Rollbar (https://rollbar.com) have been at this as well for the past 3 years, and I'm still stunned from time to time at how useful exception monitoring is.

For anyone looking to implement something like Haystack, definitely give Rollbar a look -- you'll find most of these capabilities there (and much more). And if any of the Haystack or GitHub devs have had a chance to play around with both, I'd be really interested to hear any feedback; drop me a line (email in profile).

jarus• 11 years ago

I can recommend Sentry which is open source and works great since many years!

https://github.com/getsentry/sentry or hosted https://getsentry.com/

gargarplex2• 11 years ago

Lovely how the copy on their homepage is "Shit happens — Be on top of it." Trust me, that same organizational/cultural ethos applies to customer service, wherein I dealt with petulant, arrogant and occasionally wrong 20 somethings.

gargarplex2• 11 years ago

The support rep engineer was truculent. Petulant was the wrong word.

zeeg3• 11 years ago

Always great to see the love for Sentry! We have definitely looked for inspiration from workflows in places like GitHub over the years. I recall when we first built Sentry and there wasn't really anything else like it out there. Probably around the same time Haystack was conceived, and the idea was starting to gain popularity in the Ruby community. It's amazing to see how much things have changed in the last almost-decade.

mdeeks• 11 years ago

Seconded! We use it extensively here. Really really valuable in our early days when we were first launching. As far as I'm concerned hooking into Sentry (or things like it) is a requirement for shipping and running a quality service. I even hook my cron jobs into it.

joshdotsmith• 11 years ago

We just interviewed Sentry founder David Cramer about exactly this topic: http://talkingcode.com/podcast/episode-10-david-cramer/

stympy• 11 years ago

I have to chime in and offer up Honeybadger (https://www.honeybadger.io) as a hosted option for monitoring, since I'm a co-founder. :)

Our goal is to make dealing with errors so awesome that you'll want to get more of them. :)

Linell• 11 years ago

I'm a Honeybadger user and love it! Extremely useful.

bra-ket• 11 years ago

what are some Javascript libraries that allow creating such pretty dashboards?

zeeg3• 11 years ago

My personal favorite is Highcharts, but it's a paid solution. Otherwise generally people build things on top of D3 these days. jQuery's Flot is also an easy entry tool for charts, but it's not as fancy out of the box.

roytomeij• 11 years ago

Dropping this here: https://appsignal.com — looks great, works even better. And it's one of the few tools that treats exceptions and performance both as first class citizens (full disclosure: I'm a co-founder).

RA_Fisher• 11 years ago

Rather than take a z-score, which is symmetrical, one idea for a small improvement might be to use a critical value from a chi-squared distribution. Since it's asymmetric, it'll probably fit the distribution better and offer a higher signal-to-noise ratio.

EDIT: or actually, since this is a count, might as well go straight for the poisson distribution or negative binomial -- same idea.

fundead• 11 years ago

Looks like a top notch implementation that's had some serious effort put into it for solving their unique pain points around deploys and monitoring. My colleagues and I at Raygun (https://raygun.io) have created an error tracking service that from an overview ticks the boxes that Haystack supports, and a whole bunch more besides for dev teams who don't happen to be GitHub :)

Tools like these become pretty invaluable when releasing code into the wild, and aside from lifecycle benefits like greatly reducing the cost of the maintenance phase, an error tracker can also alert and hand the data needed to fix the bug right to the person who can, greatly increasing dev sanity (no more relying on support ticket ping pong, and removes the onus on the end user to report the bug). Also, surfacing the attack vectors and the frequency of which adversaries attack code exposed to the web is rather eye opening.

ersoft• 11 years ago

Also, there is Opbeat (https://opbeat.com), which also does release/deploy tracking (you can see which errors come from/after which release).

spo81rty• 11 years ago

I would suggest looking at Stackify. We track exceptions similar to all the other mentioned tools in this thread, but we also provide log management. Having exceptions & logs together is a lot more powerful.

For example, we can show you all of your logging statements that happened before and after the error on the same web request.

http://stackify.com/errors-log-management/

-Founder of Stackify

marcinkuzminski• 11 years ago

There's also http://appenlight.com which has some really unique features, like custom graphs and performance monitoring. Also written in python :)

Exception Monitoring and Response

Related Stories