Prometheus: An open-source service monitoring system and time series database

http://prometheus.io/

by jjwiseman • 11 years ago

287 109 11 years ago

prometheus.io

ggambetta• 11 years ago

"Those who cannot remember the Borgmon are doomed to repeat it" ;)

Just kidding, this is looking really good, I hope to get some hands-on experience with it soon.

chris_va• 11 years ago

After seeing the Prometheus query language, I opened the comments looking for this quote :).

thesnider• 11 years ago

After working at a job that had a horrible patchwork of monitoring techniques (including at least two in-house systems), I was desperately pining for borgmon, actually. Never thought those words would come out of my mouth.

This does seem to have addressed at least a couple of the issues with that system, in that its config language is sane, and its scrape format is well-defined and typed.

ddorian43• 11 years ago

Announced here:

https://developers.soundcloud.com/blog/prometheus-monitoring...

bbrazil24• 11 years ago

The SoundCloud announcement is a great overview, there's also a series of blog posts I did on http://www.boxever.com/tag/monitoring going into more depth with end-to-end examples.

kanwisher• 11 years ago

Would be interesting how this compares to InfluxDb

sagichmal2• 11 years ago

http://prometheus.io/docs/introduction/comparison/#prometheu...

perlgeek2• 11 years ago

This looks very interesting.

From http://prometheus.io/docs/introduction/getting_started/

Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.

I wonder if we'll see some plugins that allow data collection via snmp or nagios monitoring scripts or so. That would make it much easier to switch large existing monitoring systems over to prometheus.

sagichmal2• 11 years ago

See Exporters for third-party systems[0] and Pushing metrics[1].

[0] http://prometheus.io/docs/instrumenting/exporters [1] http://prometheus.io/docs/instrumenting/pushing

bbrazil24• 11 years ago

(One of the authors here)

Just last night I wrote the https://github.com/prometheus/collectd_exporter, which you could do SNMP with. I do plan on writing a purpose-designed SNMP exporter in the next a few months to monitor my home network, if someone else doesn't get there first.

perlgeek2• 11 years ago

Awesome. I wondered if I should mention collectd as a possible source of data, and now you've already made it available.

discordianfish2• 11 years ago

I've wrote a exporter for Docker, so if you have a Docker based infrastructure you get tons of metrics by using it: http://5pi.de/2015/01/26/monitor-docker-containers-with-prom...

secure• 11 years ago

I used to use InfluxDB + a custom program to scrape HTTP endpoints and insert them into InfluxDB before.

After playing around with Prometheus for a day or so, I’m convinced I need to switch to Prometheus :). The query language is so much better than what InfluxDB and others provide.

jrv16• 11 years ago

(Prometheus author here)

Thanks, that's awesome to hear! Feel free to also join us on #prometheus on freenode or our mailing list: https://groups.google.com/forum/#!forum/prometheus-developer...

reinhardt1053• 11 years ago

Promdash, the dashboard builder for Prometheus, is written in Ruby on Rails https://github.com/prometheus/promdash

e12e• 11 years ago

So... how does this compare to http://riemann.io/ ? I just re-discovered riemann... and was thinking of pairing it with logstash and have a go. It would seem prometheus does something... similar?

bbrazil24• 11 years ago

From my look at Riemann, it seems more aimed as an event monitoring system than a time-series monitoring system. You can (and many do) use Riemann as time-series monitoring system, my understanding is that Prometheus is a bit better for multi-dimensional labels.

I could see Riemann being used as an alert manager on top of Prometheus, handling all the logic around de-duping of alerts and notification. Prometheus's own alert manager is considered experimental.

brown9-2• 11 years ago

Riemann has no concept of persistence, unless you forward events to some other system.

corford• 11 years ago

This looks great! Is an official Python client library on the roadmap?

bbrazil24• 11 years ago

Yes, I've got something working at the moment. It needs cleanup, docs, unittests etc.

If you want to help out, it's up at https://github.com/brian-brazil/client_python

clarkevans3• 11 years ago

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage.

I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metric server) model.

bbrazil24• 11 years ago

(One of the Prometheus authors here)

My understanding is that InfluxDB will work for your use case, as it's all push.

I see the pushgateway, but it seems deliberately not a centralized storage.

Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs.

May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics system, that'd push to the pushgateway and Prometheus would then scrape. Not my preferred way of doing things, but some distributed processing systems need that approach when work assigment is dynamic and opaque across slaves.

clarkevans3• 11 years ago

Each node in our network has it's own access control permissions and organizational owner. It's easy enough to provide a centralized service with an end-point to push statistics from each node. In this case, organizational participation is "distributed".

A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for security reasons, that's not going to happen.

Rapzid2• 11 years ago

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

clarkevans3• 11 years ago

What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.

bbrazil24• 11 years ago

We favour pull, but we don't force it. The collectd exporter is an integration with a push-based system for example.

My full thoughts are up at http://www.boxever.com/push-vs-pull-for-monitoring The short version is that I consider pull slightly better, but not majorly so. Push is more difficult to work with as you scale up, but it is doable.

jrv16• 11 years ago

(Prometheus author here)

Yeah, I did that benchmark with 11x overhead for storing typical Prometheus metrics in InfluxDB in March of 2014. Not sure if anything has changed conceptually since then, but if anyone can point out any flaws in my reasoning, that'd be interesting:

https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9Bu...

pauldix4• 11 years ago

InfluxDB CEO here. With the current version of InfluxDB (0.8.8), metadata should be encoded in the name of the series, which means you don't repeat it.

In the next version of InfluxDB (0.9.0), you can encode metadata as tags and it gets converted to a single id.

With either of those schemes you should see much better numbers on storage.

jrv16• 11 years ago

(initial Prometheus author here)

I wouldn't totally rule out the pushgateway for this use case. If we decided to implement a metrics timeout in the pushgateway (https://github.com/prometheus/pushgateway/issues/19), this would also take care of stale metrics from targets that are down or decommissioned. The pushing clients would probably even want to set a client-side timestamp in that case, as they are expected to be pushing regularly enough for the timestamp to not become stale (currently Prometheus considers time series stale that are >5 minutes old by default, see also the "Improved staleness handling" item in http://prometheus.io/docs/introduction/roadmap/.

paraboul• 11 years ago

what about druid.io?

JanezStupar• 11 years ago

Why do you need push replication?

From my experience with distributed systems push replication will get you in trouble, very soon.

Edit: I was too quick to post this question. An obvious scenario is where the client is behind a firewall. Never mind me. I am an idiot.

kordless• 11 years ago

Is the use case mostly about permissions and networking setup? I could see why a pull model wouldn't work in those cases.

tinco• 11 years ago

In your architecture I see a single monolithic database server called 'Prometheus'. Does it shard? I can't find it in the documentation. You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB?

The tech does look awesome though!

zacblazic• 11 years ago

At the bottom of http://prometheus.io/docs/introduction/comparison/ there is: "... Prometheus will be simpler to run initially, but will require explicit sharding once the capacity of a single node is exceeded."

bbrazil24• 11 years ago

Does it shard?

Currently you can manually vertically shard, and in future we may have support for some horizontal sharding for when the targets of a given job are too many to be handled by a single server. You should only hit this when you get into thousands of targets.

Our roadmap[1] includes hierarchical federation to support this use case.

You mention it's compatible with TSDB, why did you choose to implement your own backend, or is this a fork of TSDB?

Prometheus isn't based on OpenTSDB, though it has the same data model. We've a comparison[2] in the docs. The core difference is that OpenTSDB is only a database, it doesn't offer a query language, graphing, client libraries and integration with other systems.

We plan to offer OpenTSDB as a long-term storage backend for Prometheus.

[1] http://prometheus.io/docs/introduction/roadmap/ [2] http://prometheus.io/docs/introduction/comparison/#prometheu...

mhax• 11 years ago

I'm a little wary of a monolithic solutions to monitoring/graphing/time series data storage - it gives me flashbacks of nagios/zabbix ;)

I currently use a combination of sensu/graphite/grafana which allows a lot of flexability (albeit with some initial wrangling with the setup)

gambiter• 11 years ago

I'm not sure what's wrong with nagios or zabbix... I use them both in different capacities, and they are good at what they do.

Of course a piecemeal solution is more flexible, but as you said, configuration can be a beast, so many people prefer monolithic systems.

xfalcox2• 11 years ago

Guys, I've seen the libs for collecting services info, but how do I get OS level info, like load average, disk utilization, ram etc..?

I suppose that there's a simple service that we need to deploy on each server?

Any tips on this use case?

bbrazil24• 11 years ago

We support that use case, here's a guide: http://www.boxever.com/monitoring-your-machines-with-prometh...

xfalcox2• 11 years ago

Thanks from a fellow brazilian!

wyldfire• 11 years ago

That's great!

For machine monitoring Prometheus offers the Node exporter

Is it possible for the frontend to utilize data from the cron-invoked sar/sadc that already covers much of this data?

http://sebastien.godard.pagesperso-orange.fr/

bhuga3• 11 years ago

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server.

Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools that feel too big or too small; only graphite feels like it's doing one job fairly well.

Alerting is the worst of it. Nagios and all the other alerting solutions I've played with feel just a bit off. They're either doing too much or carve out a role at boundaries that aren't quite right. This results in other systems wanting to do alerting, making it tough to compare tools.

As an example, Prometheus has an alert manager under development: https://github.com/prometheus/alertmanager. Why isn't doing a great job at graphing enough of a goal? Is it a problem with the alerting tools, or is it a problem with boundaries between alerting, graphing, and notifications?

bbrazil24• 11 years ago

This results in other systems wanting to do alerting, making it tough to compare tools.

I'd think you've hit the nail on the head, every monitoring system has to do a bit of everything such as alerting, machine monitoring, and graphing; rather than focusing on just doing one thing and doing it well.

Prometheus comes with a powerful data model and query language, few existing systems support the same (e.g. Atlas and Riemann have the notion of a query language and labels) so we have to do a bit of everything to produce a coherent system.

I think a separate common alert manager would be good if we could combine, currently in practice that role isn't being done as you rely on de-duplication in a tool such as Pagerduty with support for silencing alerts not being unified.

jrv16• 11 years ago

Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain:

client app -> Prometheus server -> alerting rules -> alert handling

This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination.

If OpenTSDB-style dimensional metrics would have been more of a standard already, maybe that would be different.

bhuga3• 11 years ago

I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places.

For example, in response to the latest security hullabaloo, we recently added a Nagios check for libc being too old, and for running processes that are linked to a libc older than the one on disk. This isn't a time series check; it's just an alert condition. Polling for that alert condition isn't what Prometheus is going to be good at, so cramming it into that alert pipeline is awkward. If we used something like Prometheus, we'd have rules in two places already.

Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There's still some logic about what you're measuring in Prometheus, but not about what thresholds are alerts, or when, or for how long, or who cares. Threshold rules could all live in one place.

This isn't a knock on Prometheus, which of course supports a workflow along those lines, and seems interesting. Nor is it a knock on the prometheus alert system. I just wonder why software in the monitoring/notification/alerting space always ends up muddy. I believe the overlapping boundaries between projects are a symptom, not a cause. I keep asking myself if there's some fundamental complexity that I'm not yet appreciating.

KyleBrandt3• 11 years ago

Shameless plug. But Bosun's focus is largely on Alerting (http://bosun.org). We have an expression language built in that allows for complex rules. It leverages OpenTSDBs multi dimensional facet to make alert instantiations, but you can also change the scope by transposing your results.

It also now supports Logstash and Graphite as backends as well. The Graphite support is thanks to work at Vimeo.

Another nice thing about Bosun is you can test your alerts against time series history to see when they would have triggered so you can largely tune them before you commit them to production.

jrv16• 11 years ago

Great. Yeah, adding unit tests for alerts is something that we still need to add the capability for. But at least you can always already manually graph your alert expressions and see how they would have evaluated over time.

Interesting work on Bosun by the way! Seems like there is quite some overlap with Prometheus, but I yet have to study it in depth. Is my impression correct that OpenTSDB is a requirement, or is there any local storage component? I guess you could run OpenTSDB colocated on a single node...

mbell• 11 years ago

From the storage system docs:

which organizes sample data in chunks of constant size (1024 bytes payload). These chunks are then stored on disk in one file per time series.

That is concerning, is this going to have the same problem with disk IO that graphite does? i.e. Every metric update requires a disk IO due to this one file per metric structure.

falcolas4• 11 years ago

This was my first thought as well. Having one inode per metric measured can easily overwhelm some file systems, and the IO overhead on those disks gets silly (especially if it's not a SSD capable of constant-time writes against sparse data).

Combine with the Cacti pull model, and I think a wait-and-see attitude is the best for this for now.

beorn73• 11 years ago

We were tempted to implement our own data management in a couple of bigger files, but extensive testing of various modern file systems (XFS, ext4) resulted in the perhaps not surprising effect that those file systems are way better in managing the data than our home-grown solutions.

beorn73• 11 years ago

(Disclaimer: I'm a Prometheus author.)

Chunks are only written to that one file per time series once they are complete. Depending on their compression behavior, they will contain at least 64 samples, but usually a couple of hundreds. Even then, chunks are just queued for disk persistence. The storage layer operates completely from RAM, only has to swap in chunks if it was evicted from memory previously. Obviously, if you consistently create more sample data than your disk can write, the persist queue will back up at some point and Prometheus will throttle ingestion.

SEJeff4• 11 years ago

As a graphite maintainer, see my other post about problems with graphite:

https://news.ycombinator.com/item?id=8908423

I'm super excited about prometheus, and can't wait to get some time to see if I can make it work on my rasberry pi. That being said, I'm also going to likely eventually work on a graphite-web / graphite-api pluggable backend to use prometheus as the backend storage platform.

The more OSS metrics solutions, the better!

bbrazil24• 11 years ago

Great! We're over in #prometheus on Freenode if there's anything we can help you with.

falcolas4• 11 years ago

Before digging into converting the backend to Prometheus, you may want to have a look at their actual persistence method. What worries me is that they are still storing one metric per file, and are copying data from one file to the next to perform checkpoints or purges (which is admittedly an improvement over whisper's method of thrashing back and forth across a single file).

They have the code to do in-place method of appending chunks, but in my admittedly brief foray through the code I could not actually find any use of it.

SEJeff4• 11 years ago

The graphite backend is pluggable. It isn't converting so much as adding support for. One of my favorite backends is Cyanite which ultimately stores the data in Cassandra.

jrv16• 11 years ago

Which storage/purge strategy would you recommend? (see also this thread for more discussion about the storage: https://news.ycombinator.com/item?id=8996832)

mdeeks4• 11 years ago

A huge problem I'm having with graphite right now (which is making me look at influxdb, etc) is its inability to render graphs with lots and lots of lines. For example CPU usage across a cluster of of hundreds of machines almost always times out now. I essentially am graphing this: system.frontend-.cpu-0.cpu-used Where "frontend-" expands to 200 or so machines. I'm not entirely sure where the bottleneck is here. Would love to know if you have ideas. Is this a limitation of graphite-web itself?

I have large graphite install of 20+ graphite carbon nodes running on SSDs and three additional graphite-web instances in front generating graphs. Ingesting something like 1 million metrics/min.

Also I didn't realize there were still graphite maintainers (seriously. not trolling). There hasn't been a release of graphite in well over a year. I assumed it was dead by now. Any idea when we'll get a fresh release?

SEJeff4• 11 years ago

We are in the final stages of the last 0.9.x release, 0.9.13. From then on, we're going to be making some more noticeable changes that break some backwards compat to make the project a lot more pleasant.

Note that 0.9.13 is almost ready to be cut: https://github.com/graphite-project/graphite-web/commit/7862...

https://github.com/graphite-project/carbon/commit/e69e1eb59a...

https://github.com/graphite-project/whisper/commit/19ab78ad6...

Anything in the master branch is what will be in 0.10.0 when we're ready to cut that. I think we'll spend some more cycles in 0.10.x focusing on non-carbon / non-whisper / non-ceres backends that should allow much better scalability. Some of these include cassandra, riak, etc.

For it timing out, it is a matter of general sysadmin spleunking to figure out what is wrong. It could be IO on your carbon caches, or CPU in your render servers (where it uses cairo). I'm a HUGE fan of grafana for doing 100% of the dashboards and only using graphite-web to spit out json, or alternatively to use graphite-api.

Take a look at the maxDataPoints argument to see if that will help your graphs to not timeout however.

bbrazil24• 11 years ago

For comparison I tried out >1k cpu plots in Prometheus on a m1.large with 2xHDDs. It took 20s with a cold cache.

XorNot• 11 years ago

Huh, how fortuitous. I've been looking for this exact type of thing and HN gives me a great starting place to evaluate.

paulasmuth• 11 years ago

Shameless plug: This looks quite similar to FnordMetric, which also supports labels/multi dimensional time series, is StatsD wire compatible and supports SQL as a query language (so you won't have to learn yet another DSL)

simple103• 11 years ago

Looks really promising for smaller clusters. However, the pull/scraping model for stats could be problematic for larger scale.

I've been experimenting with metrics collection using heka (node) -> amqp -> heka (aggregator) -> influxdb -> grafana. It works extremely well and scales nicely but requires writing lua code for anomaly detection and alerts – good or bad depending on your preference.

I highly recommend considering Heka[1] for shipping logs to both ElasticSearch and InfluxDB if you need more scale and flexibility than Prometheus currently provides.

[1] https://github.com/mozilla-services/heka

bbrazil24• 11 years ago

However, the pull/scraping model for stats could be problematic for larger scale.

From experience of similar systems at massive scale, I expect no scaling problems with pulling in and of itself. Indeed, there's some tactical operational options you get with pull that you don't have with push. See http://www.boxever.com/push-vs-pull-for-monitoring for my general thoughts on the issue.

InfluxDB

InfluxDB seems best suited for event logging rather than systems monitoring. See also http://prometheus.io/docs/introduction/comparison/#prometheu...

simple103• 11 years ago

Good point on push-vs-pull. I'm biased towards push because of microservices that behave like batch jobs. In effect, I'm using AMQP in a similar way as the Prometheus pushgateway.

Agreed that InfluxDB is suited for event logging out of the box, but the March 2014 comparison of Influx is outdated IMO.

I'm using Heka to send numeric time series data to Influx and full logs to ElasticSearch. It's possible to send full logs to non-clustered Influx in 0.8, but it's useful to split out concerns to different backends.

I also like that Influx 0.9 dropped LevelDB support for BoltDB. There will be more opportunity for performance enhancements.

0xdeadbeefbabe• 11 years ago

While monitoring is obviously useful, I'm not understanding the obvious importance of a time series database. Can you collect enough measurements for the time series database to be useful? I worry that I would have lots of metrics to backup my wrong conclusions. I also worry that so much irrelevant data would drown out the relevant stuff, and cause the humans to ignore the system in time. I work with computers and servers, and not airplanes or trains.

bbrazil24• 11 years ago

Can you collect enough measurements for the time series database to be useful?

Yes, instrument everything. See http://prometheus.io/docs/practices/instrumentation/#how-to-...

I worry that I would have lots of metrics to backup my wrong conclusions.

This is not so much a problem with time series as a question of epistemology. Well chosen consoles will help your initial analysis, and after that it's down to correct application of the scientific method.

I also worry that so much irrelevant data would drown out the relevant stuff

I've seen many attempts by smart people to try and do automatic correlation of time series to aid debugging. It's never gotten out of the toy stage, as there is too much noise. You need to understand your metrics in order to use them.

rgj• 11 years ago

Is it me or is it impossible to navigate the documentation on an iPad?

jrv16• 11 years ago

I don't own any Apple devices, so I can't test, but it works well on my Android phone (being Bootstrap and using responsive design). The top menu collapses into a button with three horizontal bars which shows the menu upon click. The documentation-specific navigation is always displayed expanded when in the docs section of the site, but the contents are displayed underneath it. What does it look like for you?

discordianfish2• 11 years ago

On my iphone it looks good

zeus13i2• 11 years ago

After reading this thread and comparing Influx and Prometheus, I've concluded that both look promising. I was going to go with Prometheus (as it's easier to get started with), but I was really put off by the 'promdash' dashboard - it uses iframes and depends on mysql. So I'm going with InfluxDB + Grafana and I'll keep an eye out for developments.

bbrazil24• 11 years ago

You may want to look at the console templates, as they don't have any dependencies:

http://prometheus.io/docs/visualization/consoles/

jrv16• 11 years ago

The only time where PromDash uses iframes is when you specifically add a "frame" (iframe) widget to your dashboard to embed arbitrary web content. Native Prometheus graphs and pie charts don't use iframes at all.

Some kind of SQL backend is a dependency for now, however.

zeus13i2• 11 years ago

Ah! Good to know, thanks. Somehow I missed that.

Prometheus: An open-source service monitoring system and time series database

Related Stories