nak923

Thanks, yes. It makes our life lot easier too, if they accept it, so that we do not have to constantly maintain this forked version.

Zebrium employee here: We could not use prometheus remote storage adapter because of the following issues: 1. by the time you get there, it looses lot of information (like metric type, help, out of ordered drops etc..) 2. you do not have control over when it gets sent out, as it has to come through tsdb after chunking, this adds lot of latency. 3. It is per time series based protocol, not good for sending all the samples of a scraped target as a single chunk. Sending as a single chunk, helps you to group similar metrics at the remote side and simplifies the protocol for reducing the network bandwidth.

hence we did not use remote storage adapter. We introduced a new interface that plugs into the scraper directly.

Thanks. We started with fork, so that we can contribute back to upstream in the near future.

Anil from Zebrium here. We provide an Autonomous Monitoring service that until now has used logs as the source of our anomaly detection. We wanted to augment this by correlating log anomalies with anomalies detected within a group of related metrics.

Since Prometheus is very popular in Kubernetes environments, we wanted to support discovering and scraping Prometheus targets and send those scraped metrics to our software running in the cloud for anomaly detection. Latency is important to us as we need to receive the metrics in near real time as they get scraped. We also need to preserve labels, types, and full fidelity of time stamps for anomaly detection and log correlation purposes. And we need to do all this while being as efficient as possible in sending the metrics over the wire, as this data will be going over the WAN from a user’s Kubernetes cluster to our software which is running in the cloud.

To achieve all of this, we have built and open sourced a forked instance of the Prometheus server and a new remote server that collects metrics. Quick summary of what we achieved: near real time metric updates, preserving of valuable information such as labels and types, ability to handle out of order samples, greater than 500x bandwidth reduction. You can read all about it in this blog: https://www.zebrium.com/blog/a-prometheus-fork-for-efficient... or in the github repo: https://github.com/zebrium/prometheus.

Interesting. Might need to ride lyft to go to helipad place :).