A Prometheus fork for cloud scale anomaly detection across metrics and logs

https://www.zebrium.com/blog/a-prometheus-fork-for-efficient-cloud-scale-autonomous-monitoring

by talonx • 6 years ago

82 37 6 years ago

mk_• 6 years ago

Did you raise the challenges you had with Prometheus to the dev team and community? They are usually quite responsive and open for contributions. I'd be interested in the discussion around the reasons you brought up for forking.

chaps5• 6 years ago

Not OP, but one of the biggest issues that I've had with prometheus is that doesn't support the backfilling of timeseries data. Bringing it up to the prometheus devs led to oddly indignant responses that they had no intention of supporting backfilling, because that's not what prometheus was designed for. In large production systems, like prometheus is meant for, it was a very disappointing missing feature, and I could absolutely see why someone would want to fork if nothing else, just to get away from that level of indignancy.

hnarn3• 6 years ago

I don't know if the Prometheus devs care about it being an enterprise level monitoring software, but not having any type of backfilling on the road map pretty much disqualifies it. Accurate reporting is not a nice-to-have for enterprise grade monitoring.

omeze• 6 years ago

By backfilling I assume you mean retroactively applying labels to a time series? Can you link to the thread with more context if you have it around? As a prometheus user myself at a 100+ eng org company I've never had a use case for a flow that would require editing old timeseries apart from having to restore data after a crash

icook• 6 years ago

FWIW, backfilling is on their roadmap[0].

There's also a ticket open about it[1].

[0] https://prometheus.io/docs/introduction/roadmap/#backfill-ti...

[1] https://github.com/prometheus/prometheus/issues/535

Rapzid• 6 years ago

Yeah, if that were the only reason given I would 100% be like "Ah, yep, good call".

mekster• 6 years ago

VictoriaMetrics seems to support that. I feel this is quite technically superior to other remote storages.

https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/FAQ

(Not affiliated, just impressed with the author's blog posts explaining how it's better than competitors.)

greendave2• 6 years ago

Disclosure: Zebrium employee.

We've had a great experience with prometheus - both the community and the project. Building what we did would have been immeasurably harder without them. That said, this has been a rapid process for us. We are starting to engage and see what parts of our fork they would be interested for upstream. (In particular our storage needs are a bit different, so we'll see how that fits.)

conjectures2• 6 years ago

Anyone know what statistical models Prometheus uses?

Having scanned this article, read the wikipedia entry and hit the landing page, I am none the wiser.

DeltaSigma2• 6 years ago

There isn't much in terms of modelling built into the tool itself because that kind of functionality is secondary (the tool is primarily focused on quick and efficient gathering of metrics). You can do some basic forecasting at query time (holt-winters and linear regression), but beyond that you're using one of the remote storage adapter plugins to send the metrics to a remote time series database and the apply your ML on that (like OPs service does)

https://prometheus.io/docs/prometheus/latest/querying/functi...

conjectures2• 6 years ago

Figures, a fine objective, but not what I'd imagined.

jldugger• 6 years ago

It has holtwinters as a builtin: https://prometheus.io/docs/prometheus/latest/querying/functi...

Nothing stopping folks from adding in more.

DeltaSigma2• 6 years ago

Alternative title: "We layered compression on top of the Prometheus remote storage adapter"

nak9232• 6 years ago

Zebrium employee here: We could not use prometheus remote storage adapter because of the following issues: 1. by the time you get there, it looses lot of information (like metric type, help, out of ordered drops etc..) 2. you do not have control over when it gets sent out, as it has to come through tsdb after chunking, this adds lot of latency. 3. It is per time series based protocol, not good for sending all the samples of a scraped target as a single chunk. Sending as a single chunk, helps you to group similar metrics at the remote side and simplifies the protocol for reducing the network bandwidth.

hence we did not use remote storage adapter. We introduced a new interface that plugs into the scraper directly.

sagichmal3• 6 years ago

Anomaly detection is snake oil.

jcims• 6 years ago

Datadog figured it out. It's pretty great. Obviously it doesn't see everything, but it's a very useful signal. Their anomaly correlation is even cooler.

dang• 6 years ago

It would be good to add information explaining why. Empty dismissals don't contribute much, whereas if you make a substantive case, readers can learn.

pranay01• 6 years ago

Would love to learn why do you say so?

manigandham4• 6 years ago

Yet another Prometheus/time-series backend project.

And yet again, it would be far better to just export the data from Prometheus into a distributed columnstore data warehouse like Clickhouse, MemSQL, Vertica (or other alternatives). This gives you fast SQL analysis across massive datasets, real-time updates regardless of ordering, and unlimited metadata, cardinality and overall flexibility.

Prometheus is good at scraping and metrics collection, but it's a terrible storage system.

SuperQue2• 6 years ago

Yup, and that's kinda intentional. The design is to be a monitoring system, not a generic TSDB. What it does needs to be simple, fast, and reliable so that your alerts get sent out.

The original design inspiration, borgmon, also was a terrible storage system and had an external long-term store layered on top of it.

This isn't a design flaw, it's an intentional trade off to make the core use case as bulletproof as it can be. Having seen "monitoring systems" based on something like Cassandra, aka distributed storage, is cringe inducing. The first thing to crash a the first sign of network trouble is distributed storage.

stochastimus2• 6 years ago

I agree. Prometheus is very good at what it was designed to do.

manigandham4• 6 years ago

My point is that monitoring != storage and there are plenty of great storage systems to use so there's not much reason to create another one. For some reason developers love to create home-made (time-series) databases.

stochastimus2• 6 years ago

Ze founder here. We adapted the scraper to our needs, and figured others have probably wanted to do what you're suggesting, so we decided to share it. In fact we pick up the scrape and ingest it into a distributed column store ourselves, where we use it with logs and do anomaly detection with it. I think Prometheus is pretty good at what it does. But like you're saying, depending on what you want to do with the data, sometimes a different backend is useful.

manigandham4• 6 years ago

The big innovation here is the data compression. The lack of metric type for the standard remote storage interface is a good note.

Wouldn't adding that let you switch back to the main project and lower the local storage buffer to as small as possible?

RabbitmqGuy• 6 years ago

Hi.

I would be interested in a demo of the logs AI stuff using real data. Something like, https://www.honeycomb.io/play/ would do.

Do you have such?

greendave2• 6 years ago

Disclosure: Zebrium employee.

The ability to export from prometheus to a full-featured SQL database is definitely one aspect. In addition, for scaling, efficient transport from the scraper to the data store becomes pretty important.

Beyond just compression, having a transport protocol that takes advantage of the fact that much of the data does not change across successive scrapes, and the data that does change is often incremental (e.g. counters) makes a big difference.

manigandham4• 6 years ago

The compression part is definitely great work. It would be a valuable addition to the main prometheus repo if they accept it.

A Prometheus fork for cloud scale anomaly detection across metrics and logs

Related Stories