Show HN: Pup, real-time app metrics with Statsd

http://www.datadoghq.com/2012/08/easy-app-metrics-with-pup/

by Vvick727 • 14 years ago

107 50 14 years ago

Very cool! Is it possible to use an existing StatsD installation if we already have one running? Or is it pretty much an "all or nothing" package?

olidb215• 14 years ago

Pup will work with any existing statsd client - it just replaces the node.js statsd server. In other words, you can keep all your code and apps untouched (FYI I'm not the author, but a Datadog co-founder)

scottostler• 14 years ago

Nice! And that's an awesome logo.

alq3• 14 years ago

Thanks!

jongala• 14 years ago

Double thanks :)

brokentone• 14 years ago

Excellent project! If I understand this right, it's an addition/replacement for StatsD that adds the graphing normally done in Graphite. I've tried to get the whole Graphite/StatsD stack setup before, and I could never get Graphite together quite right.

josegonzalez• 14 years ago

That's what it looks like. I'm not sure I really care much for it, other than the fact that it no longer appears to use the Graphite URL api. Personally, I'd rather have graphite-web running and then use an alternative frontend like Descartes[1] or Graphiti[2].

Still, it's pretty good that there is an alternative frontend, if nothing else but to expand the graphite ecosystem.

  [1] https://github.com/obfuscurity/descartes
  [2] https://github.com/paperlesspost/graphiti

sciurus• 14 years ago

Somewhat off-topic, but what value is there in using statsd rather than talking to carbon, the graphite backend, directly? Statsd receives metrics over udp; carbon-cache can receive them over tcp, udp, or amqp. Statsd aggregates metrics and flushed them a set intervals; if you want this, carbon-aggregator can do it.

http://graphite.readthedocs.org/en/0.9.10/carbon-daemons.htm...

achompas4• 14 years ago

Other, more experienced people can speak to this, but I'd argue that statsd has a useful ontology for classifying metrics based on what you want them to track.

If you're watching a number go up, use a statsd.Counter. Want to track request arrival rates? Use statsd.Gauges. Want to record times for request fulfillment? Use statsd.Timers.

Statsd isn't necessarily better or worse than carbon-cache via UDP, but it provides a handy solution for the above use cases.

axx3• 14 years ago

Am i understanding this right: this is a kick in the butt for services like NewRelic?

Does this always need to run on the same server as the Application? Or is it possible to run it on a different server and push the data to it?

disclaimer: i have no experience with statsd/graphite etc.

StavrosK• 14 years ago

Statsd accepts data over UDP, so you can run it on whichever server you like, and the logging is asynchronous and takes a few microseconds (last time I counted). I haven't had a chance to use it in production extensively, but I tested it a bit and liked it very much.

olidb215• 14 years ago

So: - It doesn't need to run on the same server as the application, and you can have any number of apps reporting to it. The only limitation here is that apps communicate with Pup in UDP.

- Our goal with pup was to make it super-easy to see app metrics. It is therefore much narrower in scope than services like New Relic or projects like Graphite. It's open-source, though, and you can take it anywhere you want.

- We ourselves operate a service that can consume data from Pup and other sources, and provides metrics + events aggregation and correlation, fancy graphing, alerting, etc.. You can check-it out at http://datadoghq.com

axx3• 14 years ago

So if i understand this right:

- DogStatsD is running on my Server and is collecting data. - I can use "dogstatsd-ruby" to collect data from my (i.e.) Rails App. - DogStatsD reports Stats to your Service Datadog HQ (optional) - Pup is a small version of Datadog HQ that is running locally on my Server and connects directly to DogStatsD. - If i feel Pup doen't satisfy my needs, i can switch from Pup to Datadog HQ, get more functions and you my money? :)

right?

amix• 14 years ago

This seems to be basically a collector that sends data to Datalog (a NON open-source service that costs $15/host/month). I recommend setting up Grahite+statsd (it's truly amazing and open-source realtime metrics setup).

olidb215• 14 years ago

This was developed for Datadog, yes. But Pup works fully stand-alone and is completely open-source - so you can take whatever you want / need from it.

Pup has a narrower scope than Graphite (which we also use and love), but optimizes for shortest time to app metrics viz.

(Oh and Datadog itself is all free up to 5 hosts - but remains completely optional)

achompas4• 14 years ago

> but optimizes for shortest time to app metrics viz.

I'm not sure I understand what "shortest time to app metrics viz" means. Are you referring to the time Pup takes to create charts? Or the lowest time frequency (minutes, seconds etc.) it can handle?

Graphite defaults to reporting at 1-minute intervals, and can report at a finer granularity (1 sec. intervals) if you tweak a few settings and set up a cluster. I'm just wondering how Pup differs in this respect.

treeder• 14 years ago

@amix, you clearly didn't read the post.

bgnm2000• 14 years ago

Too purpley.

blaines• 14 years ago

This is why the guys at DataDog are awesome. Thanks!

FreshCode• 14 years ago

Love the Pup branding! Awesome logo.

achompas4• 14 years ago

Very cool app! I'm using Graphite a lot at work these days, so it's cool to see an alternative from a team based in NY.

otterley4• 14 years ago

This looks potentially interesting, but I can't really evaluate it without thorough and accurate documentation. It would be nice if there were more than just instructions for downloading it and a trivial use case. Without it, your work's not done.

katabatic• 14 years ago

I was able to get in running against an existing statsd installation trivially, and explore it to my heart's content on my existing metrics. And I can look at the source - what more do you want? That was the easiest 3rd-party dashboard setup I've ever experienced.

olidb215• 14 years ago

Thanks katabatic - reducing friction was exactly what we were going for.

otterley4• 14 years ago

If your needs are trivial, then it's probably adequate. Not everyone's needs are, though. I'm trying to evaluate it in the context of needing an enterprise-grade dashboard solution, with tens of thousands of metrics across thousands of hosts, and robust filtering/aggregation capability.

To my knowledge nobody has really solved this problem yet in a satisfactory way, outside some closed-source solutions at big internet companies.

jondot• 14 years ago

Nice project, I love the UI.

I solved the same problem in another way with:

* collectd

* collectd to graphite plugin (for server-level metrics, from all your servers)

* graphite

* statsd (for custom app metrics over UDP)

* graphene - my own UI toolkit for graphite, also based on D3.js (http://github.com/jondot/graphene)

All in all, I got a better, more robust and detailed solution using these components. Using Chef, the time to set it up is very minimal.

olidb215• 14 years ago

Nice work with Graphene, indeed.

michaelmior4• 14 years ago

I love the one line install. The solution certainly won't be perfect for everyone but having such a pain-free way to test it out is awesome.

alq3• 14 years ago

Thanks! We really had the get-started-quickly scenario in mind.

michaelmior4• 14 years ago

One thing I find disappointing is that you can't delete severs once they've been added. Especially since they count towards your limit. Unless I'm missing something?

olidb215• 14 years ago

I should add that Team @datadoghq is on IRC in #datadog on irc.freenode.net

Don't be shy if you have questions when running pup.

dbyrd• 14 years ago

Square has a good solution built on node & mongo called cube (http://square.github.com/cube/).

To build custom UIs they wrote javascript libraries Crossfilter (http://square.github.com/crossfilter/) & Cubism (http://square.github.com/cubism/)

instakill• 14 years ago

What are some potential use cases for pup?

alq3• 14 years ago

A no-hassle, real-time display of custom metrics for developers: 1 command to run and you can watch your app in action, as long as you instrument it with a statsd client (pretty much standard these days).

So, no logs to parse to get any app metric, everything already graphed for you in real-time, quasi-nil setup.

We chose statsd because it means that you don't have to change your code to get it to production. You can then rely on pup for display (but no historical data), or on your internal tool chain (e.g. graphite et al.) or give our service (http://www.datadoghq.com) a try if you don't want to set your own stuff up.

hcarvalhoalves• 14 years ago

Does it only integrate with statsd, or could it aggregate metrics from collectd, for example? This could be an alternative to Graphite for graphing metrics because Graphite is a pain in the ass to setup (no proper installer), comes with a lot of bloat and is in a semi-broken state regarding features right now.

olidb215• 14 years ago

Pup will graph all the metrics Datadog does - it already collects OS metrics, and comes with a set of plug-ins for connecting to common apps, and it wouldn't be very difficult to connect collectd to it.

Pup's feature-set is limited today, though, compared to graphite. You may also check-out the full http://datadoghq.com service for more. It's SaaS, and free up to 5 servers.

Show HN: Pup, real-time app metrics with Statsd

Related Stories