Ingest OpenTelemetry metrics with Prometheus natively

https://last9.io/blog/native-support-for-opentelemetry-metrics-in-prometheus/

by donutshop • 3 years ago

90 32 3 years ago

gouthamve4• 3 years ago

Hi! Author of the PR here. As a project, Prometheus would like to become more OTel native and this is only the first of changes that are coming.

I’d be happy to answer any questions you have.

ranting-moth2• 3 years ago

Does it still apply that if my Prometheus goes down or network glitches then metrics for that period is lost forever?

gouthamve4• 3 years ago

Unfortunately, yes. The OTel Collector has plans to implement a WAL for the OTLP exporter and when it does, you should be resilient to upstream temporarily having issues.

kbouck• 3 years ago

What are the merits of the prometheus approach versus one where events/metrics and their original timestamps can be preserved, stored (during temporary outages), then forwarded to backends when connectivity is reestablished?

worserer2• 3 years ago

Do you have a roadmap of the changes you are looking to bring? Any changes to Otel you’re looking to add as well?

bitcharmer2• 3 years ago

We've been using influx with much success. I just don't think Prometheus'es pull model is the right one for metrics especially in isolated sites like a DC. Has anyone successfully migrated from influx to Prometheus? If so, why did you do that? What's better now?

IggleSniggle• 3 years ago

I thought Prometheus was a pull or push model? Although granted, I've spent very little time with it

JimBlackwood• 3 years ago

Prometheus will always pull metrics from a metrics exposing endpoint. However, Prometheus can then push metrics to anything that has the proper integrations for remote writing[0]. So you could run Prometheus in Agent Mode [1] in your DC to ingest metrics and push it to some central location.

[0] https://prometheus.io/docs/operating/integrations/#remote-en...

[1] https://prometheus.io/blog/2021/11/16/agent/

l33tman• 3 years ago

I've been using telegraf + influxdb + grafana for my last projects, never really had to tweak anything after uncommenting the right sections in the telegraf.conf.. is prometheus and associated tools kind of an alternative to that stack?

valyala3• 3 years ago

Yes - take a look at VictoriaMetrics. https://docs.victoriametrics.com/guides/migrate-from-influx....

valyala3• 3 years ago

VictoriaMetrics has many users, who successfully migrated from InfluxDB. It supports data ingestion via Influx line protocol, so you can continue using Telegraf and sending the collected metrics to VictoriaMetrics instead of InfluxDB. You get the following benefits after the migration from InfluxDB to VictoriaMetrics:

- Reduced memory usage by up to 10x [1].

- Reduced disk space usage.

- Higher query performance.

- Better query language than InfluxQL and Flux for typical queries over collected metrics [2].

- Compatibility with Prometheus ecosystem.

See also InfluxDB -> VictoriaMetrics migration guide [3].

[1] https://valyala.medium.com/insert-benchmarks-with-inch-influ...

[2] https://docs.victoriametrics.com/MetricsQL.html

[3] https://docs.victoriametrics.com/guides/migrate-from-influx....

bitcharmer2• 3 years ago

Awesome, does victoria work with grafana?

methou• 3 years ago

So no need for Jaegermeisters at work anymore!

wdb2• 3 years ago

Nice, do I understand it correctly that this would mean there is a straight forward way to let Prometheus ingest the new histogram type without needing any new daemons (like otel-collector)?

Currently, it doesn't appear the text format for pull metrics doesn't appear to support it.

bboreham3• 3 years ago

The protobuf format supports native histograms. That’s the most straightforward way, if you have a client library for it. Go does, for instance.

OpenTelemetry is push, so if you need pull and no new daemons this PR doesn’t help you.

wdb2• 3 years ago

Yeah, I have been reading up a bit. Only it's not clear to me if the protobuf format and text format can be mixed?

Can one service A use text-format and service B use protobuf both scrapped by the same Prometheus sidecar?

thefrozenone• 3 years ago

I've had an interesting time transitioning our project from OpenCensus to OpenTelemetry now that the former is EOL'd. We use the otel stackdriver output. Anyone have a refernce comparison between GCP cloud metrics vs. a prometheus monitoring stack?

t-spawn2• 3 years ago

I did use stackdriver for quite a while before I moved to Mimir. TBH its great that you are still sticking to opentelemetry. Stackdriver as metric storage is not even a wise option in todays world give there are some really good TSDB providers SaaS or otherwise that would do a much better job.

I moved away because of 2 primary reasons

1. The cost of stackdriver can add up with large-scale deployments or high-frequency metrics. It's essential to monitor and control usage to avoid unexpected billing.

2. I have experienced delays in metric updates, specifically at high frequency data. While the delays are usually minimal, they may not be ideal for some real-time monitoring use cases. FYI GCP on its own resources makes metrics available after 210s so you are always behind.

Going the TSDB route to reliably run storage has worked for me.

Also if this helps https://last9.io/blog/time-series-database-comparison/

valyala3• 3 years ago

Prometheus was known as a monitoring system, which promotes pull model over push model for metrics' collection. E.g. it is configured to discover scrape targets and then to scrape metrics from them at regular intervals - https://www.robustperception.io/its-easy-to-convert-pull-to-... .

It is interesting to see whether Prometheus will be transformed to multi-model (pull+push) monitoring system after the addition of Opentelemetry protocol.

P.S. VictoriaMetrics, the Prometheus-like monitoring system I work on, also gained support for OpenTelemetry data ingestion in the release v1.92.0. https://docs.victoriametrics.com/CHANGELOG.html

Ingest OpenTelemetry metrics with Prometheus natively

Related Stories