Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

https://serhack.me/articles/monitoring-infrastructure-grafana-influxdb-connectd/

by serhack_ • 6 years ago

385 288 6 years ago

This is exactly what I have been planning to do for monitoring my two Raspberry pis. I am still debating on the metrics collector though. My workplace uses monitord for AINT and telegraf for wavefront. I have no idea how well does collectord works.

overcast2• 6 years ago

Really, REALLY tried to love InfluxDB. But its systems requirements, performance, and features are poor compared to things like TimescaleDB.

viraptor7• 6 years ago

Does any of those matter for small-scale monitoring (say <= 10 hosts)? I've got influx sitting pretty much idle in that kind of environments and any data preprocessing / collation in grafana works just fine.

overcast2• 6 years ago

Sure, if you're really not doing anything with it, then any database would work fine for you.

dahfizz• 6 years ago

Arguably system requirements matter more for small scale deployments. I shouldn't need a server with multiple SSD volumes and 8GB+ ram just to monitor a couple raspberry pi's.

EDIT: okay, I get it. You don't need 8GB ram. I was just going by the hardware requirements in their docs: https://docs.influxdata.com/influxdb/v1.8/introduction/insta...

candiddevmike• 6 years ago

Influx is really shitting the bed with how they're handling the InfluxDB 2.0 release. The docs are a mess, and the migration tool seems like the result of a weekend hackathon. They're leaving a lot of customers with long term metrics in a tough spot.

If you're thinking about using Influx for long term data storage, look elsewhere. The company continuously burns customer goodwill by going against the grain, and bucking the ecosystem by trying to control the entire stack.

rbetts2• 6 years ago

(I'm the VPE at Influxdata).

I appreciate this sentiment. We've been focused on a building a SaaS version of Influxdata and are committed to a paired open source version of that. The open source version has been lagging as we work on the SaaS side.

However, we are committed to shipping a GA version of the OSS 2.0 stack around the end of Q3 that offers an in-place data migration capability from 1.x OSS.

We've spoken about this publicly in other forums. You can google "influxdays London talks" to hear Paul Dix (CTO/Founder) talk more about our OSS plans.

mhall11920• 6 years ago

Just to add to what Ryan already said, we don't plan on dropping support for InfluxDB 1.x anytime soon, and will continue to improve the migration path from 1.x to 2.0, so you won't have to upgrade until it's right for you.

Now of course our goal is to help everyone upgrade to 2.0 and beyond, but we know that we made a lot of changes and improvements in 2.0, this isn't a minor upgrade. We will focus first on what we need to do to help 80% of our users upgraded, then the next 80%, and so on, until we've got you all covered.

Meanwhile new users can benefit from starting off on 2.0 as soon as it's available (or get the beta which is already out) and not have to wait.

(Source: I'm the Community Manager for InfluxData)

mbell• 6 years ago

We had major issues with scaling InfluxDB. We use clickhouse (graphite table engine) now and it is more than order of magnitude more resource efficient.

goliatone• 6 years ago

Im on the same boat, I tried very hard. Went through a few version upgrades with data incompatibility and other issues. Used it for personal projects as well as for different projects at work. I truly regret ever recommending it to different teams at work.

I ended up moving away from it to TimescaleDB and I’m pretty happy so far.

julioo3• 6 years ago

What about storage? We are running influxdb and we are looking for alternative. But a point where Influx is good is storage.

tgtweak2• 6 years ago

You mean storage efficiency? Seems that unless the aggregation is happening before being stored, it would be unfair to compare influxdb with other databases that are tasked with storing per-record granularity.

k-rus4• 6 years ago

TimescaleDB provides such community features as compression, which allows to save space a lot, and continuous aggregates, which gives performance and save space if used together with retentions.

mekster6• 6 years ago

Tried VictoriaMetrics?

The author compares with other storage engines comprehensively from time to time.

https://medium.com/@valyala

RedShift12• 6 years ago

InfluxDB is pretty good if you don't need to do any advanced querying like grouping by month or formulas. Its strong points are low diskspace footprint and very fast queries even over long periods of time.

mhall11920• 6 years ago

Have you looked at Flux? You can do some really incredible things with it, both inside of queries and also in tasks/alerts. Check out https://www.influxdata.com/blog/anomaly-detection-with-media... for an example

svd4anything• 6 years ago

Is that really true? Do you have any reference for that, I’m curious how different it is. FWIW I use InfluxDB and haven’t really thought it was a huge resource hog.

valyala12• 6 years ago

InfluxDB may require big amounts of RAM when working with high cardinality data. It works OK with up to a million of time series. Then it starts eating RAM as crazy [1]. How many time series does your InfluxDB setup contain?

[1] https://medium.com/@valyala/insert-benchmarks-with-inch-infl... .

majkinetor2• 6 years ago

I use InfluxDb on number of big gov production services, and never had a single issue with it. I use it for all metrics (applicative and infrastructure).

Its easy to deploy and its x-platform. Not sure what are all the comments here about - maybe for really huge loads which I don't have experience at, I usually use separate influxdb per service.

viraptor7• 6 years ago

Nothing against choosing this set of apps really, but I'm curious why collectd and not telegraf which does the same kind of metrics probes and is a part of the TICK stack.

bbrks• 6 years ago

As somebody new to both, are you able to elaborate on what telegraf/TICK provides that the collection of programs in the post does not?

I see lots of posts complaining about the stack in the post, but all of the alternatives posed don't really explain why.

viraptor7• 6 years ago

Telegraf and collectd do roughly the same thing. They run some plugins to get data and push the results to a given metrics sink. I asked because TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) is a known solution and a fairly standard way to add elements of monitoring to your system.

cbzbc3• 6 years ago

collectd comes packaged by default in a bunch of distributions and telegraf does not -- for basic functionality telegraf doesn't really buy anything and takes more configuration.

mhall11920• 6 years ago

`snap install telegraf` is pretty easy though

sciurus3• 6 years ago

Although I've successfully used collectd in the past, I would choose telegraf over it now. This is especially true if I'm sending data to InfluxDB,s ince it and telegraf are primarily developed by InfluxData.

Both telegraf and collectd are mature programs with a large base of well-documented plugins. However, telegraf development is more active, and I find its golang codebase much more approachable than the C codebase of collectd. Obviously, YMMV.

In general though, I wouldn't advise adopting the entire TICK stack. Chronograf is inferior to Grafana. Kapacitor has a super-steep learning curve and is being deprecated by InfluxData. TIG (telegraf, influxdb, grafana) can be a good choice. Prometheus plus Grafana is worth evaluating as well; there's arguably more momentum in that ecosystem than in InflxuDB's.

Kednicma• 6 years ago

I'm curious why they chose CollectD and not Prometheus or another pull-based ("scraping") solution. Pulling is more compositional than pushing and requires fewer machines to be touched when monitoring configuration is changed.

viraptor7• 6 years ago

Probably due to the size of deployment. I'd go to Prometheus in a larger/dynamic environment for flexibility, but I'm happy using telegraf on a home network where I configure things by hand when I'm logged into each pet machine.

mhall11920• 6 years ago

The author responded in another thread, it was just due to being familiar with CollectD already

wiradikusuma• 6 years ago

There's also https://github.com/timberio/vector

victor106• 6 years ago

+1 for vector. We moved from Logstash to Vector and we couldn't be happier. Logstash is awesome but its a memory hog.

With Vector and Toshi you can kinda (I am not sure Toshi is as mature as Elastic) use them to replace LogStash and Elastic, the missing piece is Kibana

sevagh3• 6 years ago

Have you looked into Grafana Loki for logs? If I had to redo one part of our stack, that's what I'd choose.

lma212• 6 years ago

Does Vector work with TimescaleDB? it looks quite interesting

pachico5• 6 years ago

I approached InfluxDB since it looked promising. It did actually served its purpose when it was simple and Telegraf was indeed handy. Now that I have more mature requirements I can't wait to move away from it. It gets frozen frequently, it's UI Chronograph is really rubbish, functions are very limited and managing continuous queries is tiresome.

I'm now having better results and experience storing data in ClickHouse (yes, not a timeseries dB).

From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising.

It might just be me.

willvarfar• 6 years ago

I have also had scalability and reliability issues with influx. And full of silly limitations like tagset cardinality and not being able to delete points in a specific retention policy etc. Am moving to classic rdbms and timescaledb.

pachico5• 6 years ago

Indeed, cardinality limit is a very painful aspect which blocked us since day one for certain metrics. I confess, at the time I didn't know any better. Now I wouldn't recommend it under any circumstances.

gregwebs• 6 years ago

Have you tried Victoria Metrics? It originallyl started as a ClickHouse fork for time series but is now a re-write keeping some of the same principals.

pachico5• 6 years ago

Yes, we use it as long term retention for Prometheus. I was also tempted to use it as stand-alone dB but never got to it. Do you use it as such?

mhall11920• 6 years ago

From time to time I also follow what's coming in InfluxDB 2.0 but I must confess that 16 betas in 8 months are not very promising.

Don't read too much into that, it's more a result of wanting to get testable releases out early and temporarily redirecting engineering resources to take advantage of opportunities that arose in that timeframe than anything having to do with the code of 2.0 itself.

Our Cloud 2 SaaS offering is already running the 2.0 code in production (albeit with changes to support it being deployed as a massively multi-tenant service)

pachico5• 6 years ago

Fair enough. Good luck then with 2.0. I just wish the value proposition is strong enough to inspire users like me to try it again although much has to change for that to happen. I just wish I didn't have the feeling that the free version is just a bait for the SaaS. Anyhow, as said, good luck.

second--shift• 6 years ago

I've looked at these before, and I remember a few years ago when Grafana was really starting to get big, but I guess I have a bona-fide question: Who really needs this?

I manage a small homelab infra, but also an enterprise infra at work with >1,000 endpoints to monitor, and I/we use simple shell scripts, text files, and rsync/ssh. We monitor cpu load, network load, disk/io load, all the good stuff basically. The monitor server is just a DO droplet and our collectors require zero overhead.

The specs list and setup costs in time and complexity are steep with a Grafana stack - is there any value besides just the visual? I know they have the ability to do all manner of custom plugins, dashboards, etc, but if you just care about the good stuff (uptime+performance), what does Grafana give you that rsync'ing sar data can't?

PS: we have a graphical parser of the data written using python and matplotlib. very lightweight, and we also get pretty graphs to print and give to upstairs.

Nextgrid• 6 years ago

The biggest advantage is the near-real-time aspect of this. During an outage, having live metrics is essential. Does your custom system allow you to see live metrics as they happen, or do you need to re-run your aggregation scripts every 5 minutes?

dewey4• 6 years ago

At work and in my personal projects I use a Prometheus + Grafana setup and it's very easy to set up. Not sure what you mean with the complex and setup costs for a Grafana stack?

Alerting with the Prometheus AlertManager is also pretty straight forward and I'm looking at dashboards every day to see if everything is running smoothly or tracking down what's not working well if there are any issues. Grafana dashboards are always the second thing I look at after an alert fires somewhere and it has been invaluable.

viraptor7• 6 years ago

What sar+rsync doesn't provide:

- app-specific metrics

- quick and easy way to build a number of graphs searching for correlation (how to slice the data to get results that explain issues)

- log/metrics correlation

- unified way to build alerts

- ad-hoc changes - while you're in the middle of an incident and want to get information that's just slightly different from existing, or filter out some values, or overlay a trend - how long would it take in your custom solution vs grafana?

And finally - grafana exists. Why would I write a custom graph generator from a custom data store if I can setup a collector + influx + grafana in a fraction of that time and get back more?

lifeisstillgood• 6 years ago

I love this attitude (simplest thing that can possibly work) and am trying to write book about how to run a whole syseng department in this way.

In my view grafana shines in app data collection. And what we want a lot of is that (my simplest thing that can possibly is just have a web server accepting metrics ala carbon/graphite)

So one is likely to have grafana already lying around when one does the infra monitoring - I guess your choice is now how to leverage your set up to do app monitoring ?

gjulianm• 6 years ago

I'm not experienced with the CollectD stack, but I use Prometheus + Grafana to monitor probes. My two cents:

- Fairly lightweight. Prometheus deals with quite a lot of series without much memory or CPU usage.

- Integration with a lot of applications. Prometheus lets me monitor not only the system, but other applications such as Elastic, Nginx, PostgreSQL, network drivers... Sometimes I need an extra exporter, but they tend to be very light on resources. Also, with mtail (which is again super lightweight) I can convert logs to metrics with simple regexes.

- Number of metrics. For instance, several times I needed to diagnose an outage and I need a metric that I didn't think about, and turns out that the exporter I was using did actually store it, it was just that I didn't include it in the dashboard. As an example, the default node exporter has very detailed I/O metrics, systemd collectors, network metrics... They're quite useful.

- Metric correctness. Prometheus appears to be at least decent at dealing with rate calculations and counter resets. Other monitoring systems are worse and it wasn't weird to find a 20000% CPU usage alert due to a counter reset.

- Alerts. Prometheus can generate alerts with quite a lot of flexibility, and the AlertManager is a pretty nice router for those alerts (e.g., I can receive all alerts in a separate mail, but critical alerts are also sent in a Slack channel).

- Community support. It seems the community is adopting the Prometheus format for exposing metrics, and there are packages for Python, Go and probably more languages. Also, the people who make the exporters tend to also make dashboards, so you almost always have a starting point that you can fine-tune later.

- Ease of setup. It's just YAML files, I have an Ansible role for automation but you can go with just installing one or two packages in clients and adding a line to a configuration file in the Prometheus master node.

- Ease of use. It's incredibly easy to make new graphs and dashboards with Prometheus and Grafana, no matter if they're simple or complex.

For me, the main points that make me use Prometheus (or any other monitoring config above simple scripts) is alerting and the amount of metrics. If you just need to monitor CPU load and simple stats, maybe Prometheus is too much, but it's not that hard to set up anyways.

crecker3• 6 years ago

Author here. I'll probably write another tutorial focusing on Prometheus, instead of CollectD.

Thanks for suggestion

SerHack

VectorLock• 6 years ago

It sounds like you home rolled a version of Grafana, CollectD et al. which probably took more time and effort than just installing Grafana CollectD et al.

ggregoire• 6 years ago

I think you are overestimating the time and complexity to install and set up prometheus + grafana on a box, node exporter on your hosts and copy/paste a grafana dashboard for node exporter (which is your use case).

It gets complex only when you start monitoring your apps (i.e. using a prometheus client library to generate and export custom app metrics) and create custom grafana dashboards for these metrics. Or if you need to monitor some niche technology without its own existing prometheus exporter. Then yes, you need to read the docs, think about what you need to monitor and how, write code...

ddevault3• 6 years ago

I found the software in this stack to be very bloated and difficult to maintain. Large, complicated software has a tendency to fall flat on its face when something goes wrong, and this is a domain where reliability is paramount. I wrote about a different approach (Prometheus + Alertmanager) for sourcehut, if you're curious:

https://sourcehut.org/blog/2020-07-03-how-we-monitor-our-ser...

robotmay3• 6 years ago

I have a bit of a love/hate relationship with Prometheus. At home I really like it; it was simple to set up for my needs and most of my configuration is on my server which then scrapes other machines for the data. However I find it quite frustrating at scale for work, both in its concepts (it's hard to describe but it's sort of...backwards?) and in its query performance, although that might be a side-effect of using it with Grafana and me attempting to misuse it. By contrast I think the concepts of something like TimescaleDB are easier to understand when it comes to scaling and optimising that service.

In my previous job I had a very clear use-case for not using Prometheus and did for a while use InfluxDB (it involved devices sending data from behind firewalls across many sites). I found it pretty expensive to scale and it fell over when it ran out of storage, which feels like something that should have been handled automatically considering it was a PaaS offering.

ddevault3• 6 years ago

One point of note for SourceHut's Prometheus use is that we generally don't make dashboards. I don't really like Grafana. I will sometimes use gnuplot with styx to plot graphs on an as-needed basis:

https://github.com/go-pluto/styx

This is how I made the plots in that blog post.

dewey4• 6 years ago

I guess the https://github.com/prometheus/pushgateway could help with that? As for the query performance there's a lot of things you can do with recording rules, that might help a lot with speeding up dashboards or queries.

roland35• 6 years ago

InfluxDB and Grafana worked great for us when I created a live monitoring system for a fleet of prototype test robots. It was simple to set up new data streams. We started with Graphite but switched to InfluxDB for it's flexibility (Grafana works with both!)

I would add to the guide that you need to be careful about formatting the lines into InfluxDB because where you out the space and commas determines what is indexed or not! Also data types should be specific (ie make sure you are setting integer vs float correctly).

x87678r3• 6 years ago

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

halfmatthalfcat2• 6 years ago

The question is how to do long term storage though. Something I've had a bit of trouble rationing about.

Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

dewey4• 6 years ago

I think that's what https://thanos.io/ is for.

x87678r3• 6 years ago

Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.

sevagh3• 6 years ago

I second the Thanos link, but if you want to only use Prometheus, I would look into federation:

https://www.robustperception.io/scaling-and-federating-prome...

https://prometheus.io/docs/prometheus/latest/federation/

You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.

dfinninger• 6 years ago

How about another Prometheus server in federation[0]?

It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom).

Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization.

If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config).

[0] https://prometheus.io/docs/prometheus/latest/federation/

codeduck• 6 years ago

For long term retention, look at VictoriaMetrics

deepakhj2• 6 years ago

Thanos or Cortex

speedgoose• 6 years ago

I don't. It's quite difficult to do complex queries and its query language has a few gotcha. I think it's quite good and one of the best solutions today, but I look forward to something as simple and as fast, but with a proper query language.

x87678r3• 6 years ago

Yes would be nice to be able to run sql or something more standard. I like that its just simple, one application and it just works.

valyala12• 6 years ago

PromQL from Prometheus has steep learning curve because of lack of good documentation. When you understand PromQL basics, then it is much easier to write typical queries over time series data in PromQL than in any other query languages (SQL, Flux, InfluxQL, etc.). I'd recommend reading the article about PromQL basics [1] and then feel the power of PromQL.

[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

majkinetor2• 6 years ago

You can quickly do this on Windows:

    cinst influxdb1 /Service
    cinst grafana
    start $Env:ChocolateyInstall\lib\grafana\tools\grafana-*\bin\grafana-server.exe
    git clone https://github.com/majkinetor/psinflux
    import-module ./psinflux
    1..10 | % { $x = 10*$_ + (Get-Random 10); Send-Data "test1 value=$x"; sleep 1 }

PanosJee• 6 years ago

Too many parts Netdata will make it a breeze

linsomniac2• 6 years ago

Netdata is pretty slick.

carlosdp4• 6 years ago

Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.

Hates_• 6 years ago

I'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.

dharmab2• 6 years ago

Prometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.

ciguy2• 6 years ago

This is good advice, however you will also want to make sure you have a plan to get off datadog when you grow. Datadog is one of the easiest to use and most comprehensive out of the box. But it gets really expensive as you begin to scale up and add servers, cloud accounts, services etc....

At a certain scale, rolling your own monitoring and alerting becomes cost effective again as Datadog begins to charge an arm and a leg. I've seen Datadog bills that could easily pay for 2 full time engineers.

lima2• 6 years ago

We migrated customers off DataDog for this exact reason.

Rolling a custom Prometheus / Grafana / Alertmanager setup is not hard at all, more powerful, and it's much easier to do it right from the start.

user59944618• 6 years ago

I've seen Datadog bills that could easily pay for 2 full time engineers.

Which means the companies are running thousands of hosts. So they definitely need both datadog and a full team of sysadmin/devops/SRE to handle that infrastructure.

carlosdp4• 6 years ago

On the contrary, the bigger you get, the more you need Datadog to scale. Once you grow big enough, you can make volume deals with them, you don't pay list price. I'd much rather pay them than have a single engineer have to spend any time managing infrastructure that isn't core to our product, especially since Datadog will always do it better.

Axsuul4• 6 years ago

As a counterpoint: I have been rolling my own Grafana/Prometheus for many years for my startup and it's been pretty trivial

risyachka• 6 years ago

Same here. DataDog would cost me at least 10x.

yellow_mixer2• 6 years ago

How are you running it? On AWS? What’s your infra look like.

dijit6• 6 years ago

I just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month.

No thanks. :/

rattray6• 6 years ago

Can you say something vague about your deployment scale?

cs-szazz2• 6 years ago

Wait, you're saying it costs 12x as much as your salary? That doesn't seem right... the company I'm at uses it pretty heavily and we're at about 1x of a FTE salary (and it's still super worth it)

secondcoming• 6 years ago

We're moving from Datadog to Prometheus for this reason

jakozaur• 6 years ago

DataDog per host pricing can be very expensive. Metrics are provided by many platforms. If you need logs too, you may look at Sumo Logic which got way cheaper metrics in typical use case.

Disclaimer: I work at Sumo Logic.

bpodgursky• 6 years ago

Like any SaaS dev tool, when at scale, you negotiate and pay a fraction of the list price.

It's meaningless to look at the price of Datadog@5 hosts -- at 500 or 5000, you're paying a completely detached number from the website list price, likely a small fraction.

carlosdp4• 6 years ago

It's expensive for good reason, it's the best out there.

scaryclam• 6 years ago

This is not my experience at all. We've got it setup and monitoring all sorts of things. After the initial setup, the only reason we've needed to touch it is when we've introduced new things and wanted to update the config. In fact, data dog was far more of a pain, and far less useful than prometheus has been.

burtonator• 6 years ago

I've done this in my company with about 100 servers using KairosDB + Grafana and CollectD...

https://kairosdb.github.io/

If I were to do it again I'd probably use Grafana and Prometheus

avthar3• 6 years ago

Echoing the sentiment expressed by others here, for a scalable time-series database that continues to invest in its community and plays well with others, please check out TimescaleDB.

We (I work at TimescaleDB) recently announced that multi-node TimescaleDB will be available for free, specifically as a way to keep investing in our community: https://blog.timescale.com/blog/multi-node-petabyte-scale-ti...

Today TimescaleDB outperforms InfluxDB across almost all dimensions (credit goes to our database team!), especially for high-cardinality workloads: https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-...

TimescaleDB also works with Grafana, Prometheus, Telegraf, Kafka, Apache Spark, Tableau, Django, Rails, anything that speaks SQL...

jimaek• 6 years ago

While influx is pretty bad overall it's super simple to deploy and configure unlike timescale.

It's the main reason we decided to use influx in our small team with simple enough timeseries needs

avthar3• 6 years ago

Curious what you found difficult to deploy/ configure? Is this in a self-managed context?

souldeux• 6 years ago

What's wrong with influx? I use it and like it, albeit for hobby-level projects.

heliodor• 6 years ago

For small and medium needs, InfluxDB is a delight to use. On top of that, the data exploration tool within Chronograf is one-of-a-kind when it comes to exploring your metrics.

If you are looking for a vendor to host, manage, and provide attentive engineering support, check out my company: https://HostedMetrics.com

valyala12• 6 years ago

If you want the best of both worlds, then try VictoriaMetrics. It is simple to deploy and operate and it is more resource-efficient comparing to InfluxDB. More on this, it supports data ingestion over Influx line protocol [1].

[1] https://victoriametrics.github.io/#how-to-send-data-from-inf...

aeyes3• 6 years ago

I don't consider TimescaleDB to be a serious contender as long as I need a 2000 line script to install functions and views to have something essential for time-series data like dimensions:

https://github.com/timescale/timescale-prometheus/blob/maste...

gen2205• 6 years ago

(Not affiliated with TimescaleDB, just trying to understand your critique)

TimescaleDB is a Postgres extension, which means they have to play PG's rules, which rather implies a complicated mesh of functions, views, and "internal-use-only" tables.

But once they're there, you can pretty much pretend they don't exist (until they break, of course, but this is true for everything in your software stack).

Is your complaint targeting the size of these scripts, or their contents? Because everything in there appears pretty reasonable to me.

mfreed• 6 years ago

Because TimescaleDB is a relational database for general-purpose time-series data, it can be used in a wide variety of settings.

That configuration you cite isn’t for the core TimescaleDB time-series database or internals, but to add a specific configuration and setup optimized for Prometheus data, so that users don’t need to think about it and TimescaleDB can work with the specific Prometheus data model right out of the box.

Those “scripts” automatically setup a deployment that is optimized specifically to have TimescaleDB serve as a remote backend for Prometheus without making users think about anything. Some databases would need to write a lot of specific internal code for this; TimescaleDB can enable this flexibility through configuring proper schemas, indexes, and views instead (in addition to some specific optimizations we've done to support PromQL pushdown optimizations in-database written in Rust).

This setup also isn't something that users think about. Users install our Prometheus stack with a simple helm command.

$ helm install timescale/timescale-observability

The Timescale-Prometheus connector (written in Go) sets this all up automatically and transparently to users. All you are seeing is the result of the system being open-source rather than having various close-source configuration details :)

- https://github.com/timescale/timescale-observability

- https://github.com/timescale/timescale-prometheus

bsg75• 6 years ago

There may be some extra install steps with Timescale, but that is balanced by having data in a platform accessible by a very well known API.

There are always tradeoffs, but in cases like mine, I find the basis of the platform - PostgreSQL - to provide a very useful set of features.

cevian8• 6 years ago

(TimescaleDB engineer here). This is a really unfair critique.

The project you cite is super-optimized for the Prometheus use-case and data model. TimescaleDB beats InfluxDB on performance even without these optimization. It's also not possible to optimize in this way in most other time-series databases.

These scripts also work hard to give users a UIUX experience that mimicks PromQL in a lot of ways. This isn't necessary for most projects and is a very specialized use-case. That takes up a lot of the 2000 lines you are talking about.

gshulegaard3• 6 years ago

Just wanted to say I am super impressed with the work TimescaleDB has been doing.

Previously at NGINX I was part of a team that built out a sharded timeseries database using Postgres 9.4. When I left it was ingesting ~2 TB worth of monitoring data a day (so not super large, but not trivial either).

Currently I have built out a data warehouse using Postgres 11 and Citus. Only reason I didn't use TimescaleDB was lack of multi-node support in October of last year.

I sort of view TimescaleDB as the next evolution of this style of Postgres scaling. I think in a year or so I will be very seriously looking at migrating to TimescaleDB, but for now Citus is adequate (with some rough edges) for our needs.

roskilli• 6 years ago

If you're looking at scaling monitoring timeseries data you may also wanter to consider more Availability leaning architecture (in the CAP theory sense) with respect to replication (i.e. quorum write/read replication, strictly not leader/follower - active/passive architecture) then you might also want to check out the Apache 2 project M3 and M3DB at m3db.io.

I am biased obviously as a contributor. Having said that I think it's always worth understanding active/passive type replication and the implications and see how other solutions handle this scaling and reliability problem to better understand the underlying challenges that will be faced with instance upgrades, failover and failures in a cluster.

valyala12• 6 years ago

Did you try ClickHouse? [1]

We were successfully ingesting hundreds of billions of ad serving events per day to it. It is much faster at query speed than any Postgres-based database (for instance, it may scan tens of billions of rows per second on a single node). And it scales to many nodes.

While it is possible to store monitoring data to ClickHouse, it may be non-trivial to set up. So we decided creating VictoriaMetrics [2]. It is built on design ideas from ClickHouse, so it features high performance additionally to ease of setup and operation. This is proved by publicly available case studies [3].

[1] https://clickhouse.tech/

[2] https://github.com/VictoriaMetrics/VictoriaMetrics/

[3] https://victoriametrics.github.io/CaseStudies.html

pritambaral• 6 years ago

Obligatory heads-up: parts of TimescaleDB (including the multi-node feature) come with no right-to-repair. See https://news.ycombinator.com/item?id=23274509 for more details.

akulkarni2• 6 years ago

We are currently working on revising this to make the license more open. Stay tuned :-).

(If you want to provide any early feedback, please email me at ajay (at) timescale.com)

etxm• 6 years ago

high-cardinality workloads

Does this mean if you were using it with Prometheus you could get around issues with high cardinality labels?

say_it_as_it_is2• 6 years ago

This thread is about infra monitoring, where any of the potential performance differences probably don't matter at all. If it does, show us.

I don't work for any time series database provider.

darkwater• 6 years ago

Why performances shouldn't matter in this case? The more instances, the more timeseries and the more datapoints you have, the more you care for speed when you need to query it to visualize for example WoW changes in, say, memory usage.

didip2• 6 years ago

How does multi node works with Postgres? Does TimescaleDB create its own Raft layer on top and just treat Postgres as dumb storage?

k-rus4• 6 years ago

A TimescaleDB engineer here. Current implementation of database distribution in TimescaleDB is centralised where all traffics go through an access node, which distributes the load into data nodes. The implementation uses 2PC. Abilities of PostgreSQL to generate distributed query plans are utilised together with TimescaleDB optimisations. So PostgreSQL is used not just a dumb storage :)

richardARPANET• 6 years ago

We're using TimescaleDB + Grafana for visualising sensor data for a Health Tech product. No complaints so far.

znpy4• 6 years ago

After a few years in the industry of systems engineering and administration I think that "no complaints so far" is one of the best compliments a software can receive.

ngrilly3• 6 years ago

TimescaleDB and my team is using it, but one significant drawback compared to solutions like Prometheus are the limitations of continuous aggregations (basically no joins, no order by, no window functions). That’s a problem when you want to consolidate old data.

cevian8• 6 years ago

(TimescaleDB engineer) we hear you and are working on making continuous aggregations easier to use.

For now, the recommended approach is to perform continuous_aggregates on single tables and perform joins, order by, and window when querying the materialized aggregate rather than when materializing.

This often has the added benefit of often making the materialization more general so that a wider range of queries can use it.

ngrilly3• 6 years ago

Meant "TimescaleDB is great and my team is using it"...

mekster6• 6 years ago

Can you refute the claim that TimescaleDB uses such a huge disk space that is 50 times larger than other time series database?

I see no reason to use it over VictoriaMetrics.

https://medium.com/@valyala/high-cardinality-tsdb-benchmarks...

PaulWaldman2• 6 years ago

Yes, I can refute that claim. TimescaleDB now provides built in compression.

https://docs.timescale.com/latest/using-timescaledb/compress...

douglasheriot• 6 years ago

I've been using InfluxDB, but not satisifed with limited InfluxQL, or over-complicated Flux query languages. I love Postgres so TimescaleDB looks awesome.

The main issue I've got is how to actually get data into TimescaleDB. We use telegraf right now, but the telegraf Postgres output pull request still hasn't been merged: https://github.com/influxdata/telegraf/pull/3428

Any progress on this?

akulkarni2• 6 years ago

There is a telegraf binary available here that connects to TimescaleDB: https://docs.timescale.com/latest/tutorials/telegraf-output-...

If you are looking to migrate data, then you might also want to explore this tool: https://www.outfluxdata.com/

valyala12• 6 years ago

I believe PromQL [1] and MetricsQL [2] are much better suited for typical queries over time series data than SQL, Flux or InfluxQL.

[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

[2] https://victoriametrics.github.io/MetricsQL.html

a10c• 6 years ago

One of the biggest quirks that I had bumped up against with TimescaleDB is that it's backed by a relational database.

We are a company that ingests around 90M datapoints per minute across an engineering org of around 4,000 developers. How do we scale a timeseries solution that requires an upfront schema to be defined? What if a developer wants to add a new dimension to their metrics, would that require us to perform an online table migration? Does using JSONB as a field type allow for all the desirable properties that a first-class column would?

valyala12• 6 years ago

90M datapoints per minute means 90M/60=1.5M datapoints per second. Such amounts of data may be easily handled by specialized time series databases even in a single-node setup [1].

What if a developer wants to add a new dimension to their metrics, would that require us to perform an online table migration?

Specialized time series databases usually don't need defining any schema upfront - just ingest metrics with new dimensions (labels) whenever you wish. I'm unsure whether this works with TimescaleDB.

[1] https://victoriametrics.github.io/CaseStudies.html

zytek• 6 years ago

VictoriaMetrics eats other TSDBs for breakfast.

PromQL support (with extensions) and clustered / HA mode. Great storage efficiency. Plays well for monitoring multiple k8s clusters, works great with Grafana, pretty easily deployed on k8s.

No affiliation, just a happy user.

mekster6• 6 years ago

I just don't get why VictoriaMetrics doesn't get more visibility.

Maybe they need a PR person.

valyala12• 6 years ago

Absolutely! We are working on this.

Papric0re4• 6 years ago

[Offtopic (a bit)] Lots of you are talking about metric monitoring. But do you have recommendations when it comes to (basic) security Monitoring? I would usually go for the Elastic-Stack for that purpose, especially because Kibana offers lots of features for security monitoring. But I feel like these stacks are so big and bloated. I basically need something to monitor network traffic (Flows and off-Database retention of PCAPs) and save some security logs (I'm not intending on alerting based on logs, just for retention). But being able to have a network overview, insight into current connections (including history) is a very useful thing. Can anybody recommend something, that's maybe a bit lighter than an entire Elastic-Stack?

floren3• 6 years ago

I think Gravwell (https://gravwell.io) might be what you're looking for--but I work for Gravwell so I may be biased! If I can be forgiven a short sales pitch, we've built a format-agnostic storage & querying system that easily handles raw packets, Netflow records (v5, v9, and IPFIX), collectd data, Windows event logs, and more. You can see some screenshots at https://www.gravwell.io/technology

We have a free tier which allows 2GB of data ingest per day (paid licenses are unlimited) which should be more than enough for capturing logs and flows. The resources needed to run Gravwell basically scale with how much data you put into it, but it's a lot quicker to install and set up than something like Elastic, in our opinion (https://www.gravwell.io/blog/gravwell-installed-in-2-minutes)

Edit: it's currently a bit roll-your-own, but we're really close to releasing Gravwell 4.0 which enables pre-packaged "kits" containing dashboards, queries, etc. for a variety of data types (Netflow, CoreDNS logs, and so on)

Papric0re4• 6 years ago

I don't mind you giving a small sales pitch...and maybe your product is indeed what I'm searching for. But your pricing model is instantly putting me off. Same as with Splunk, you end up with not being able to predict your cost and paying way too much. Tell me when you fix that and I might be interested ;)

Edit: Sorry, I was misreading your comment. Premium is unlimited...I will look into it, thanks. :)

iudqnolq• 6 years ago

When you say

Gravwell is developed and maintained by engineers expert in security and obsessed with high performance. Therefore our codebase is 100% proprietary and does not rely on open source software. We love open source, but we love our customers and their peace of mind a lot more!

does that mean you've even rolled your own webserver? Programming language?

user59944618• 6 years ago

If you want to do logs, you can use graylog or kibana, both using elasticsearch for storage. This allows to find what was connecting where at some point in time (HTTP request logs and database connection logs).

If you want to graph connections from service to service in real time. I've actually never found anything that was capable of doing that, not even paid software.

tilolebo2• 6 years ago

Grafana together with Loki would be a good match for you

retzkek• 6 years ago

Maybe Loki [1] meets your needs? It lacks the analytics abilities of Elastic (e.g. what's the average response time [2]) but is much simpler to setup and use for jog aggregations, and has a pretty powerful query language for digging through and graphing log statistics (e.g. how many errors have been logged per hour). It's mainly being developed by Grafana Labs, so there's great integration in Grafana.

[1] https://grafana.com/oss/loki/

[2] I'd argue this sort of thing should be published as a metric anyways so you don't have to pull it out of the logs

Papric0re4• 6 years ago

Tanks for the tip. But I would still need some storage and Data Shipper, right? Or is Loki also taking care of storage?

mavam2• 6 years ago

(Disclaimer: CEO & founder of Tenzir)

We at Tenzir are developing VAST for this purpose: https://github.com/tenzir/vast. It's still very early stage, but if you're up for trying something new, a lean and modern C++ architecture, BSD-license open-source style, you may want to give it a spin. The docs are over at https://docs.tenzir.com/vast.

It supports full PCAP, NetFlow, and logs from major security tools. There is CLI and Python bindings. The Apache Arrow bridge offer a high-bandwidth output path into other downstream analytics tools.

tgtweak2• 6 years ago

Looking for a similar post with APM using opentracing !

rattray6• 6 years ago

I've only ever used third party monitoring tools, but hope to set up a startup again soon and want to do OSS if I can.

Can anyone comment on Prometheus vs Timescale? What are the tradeoffs? Or would I use Prometheus on top of Timescale?

k-rus4• 6 years ago

You can use Prometheus on top of TimescaleDB. Timescale builds connector and entire workflow to run Prometheus on top of TimescaleDB and support Grafana in flexible way. Sorry for the promo :) check for details in https://github.com/timescale/timescale-prometheus

rattray6• 6 years ago

Thanks!

cevian8• 6 years ago

(Timescale-Prometheus team lead here) I'd suggest using Timescale-Prometheus to connect Prometheus with TimescaleDB.

Repo: https://github.com/timescale/timescale-prometheus Design Doc: https://tsdb.co/prom-design-doc

valyala12• 6 years ago

Prometheus is a monitoring system (it collects metrics and evaluates alerts on them), while TimescaleDB is a database. While it is possible to use TimescaleDB as a remote storage for Prometheus, I'd recommend taking a look at other remote storage integrations as well [1]. Some of them natively support PromQL (like VictoriaMetrics, m3db, Cortex), others may be easier to setup and operate. For instance, VictoriaMetrics works out of the box without complex configuration. Disclaimer: I work for VictoriaMetrics :).

[1] https://prometheus.io/docs/operating/integrations/#remote-en...

aprdm3• 6 years ago

As mentioned by others I have found a Prometheus + Grafana setup to be pretty straight forward to install and maintain. It also aligns with the cloud native foundation which means that this stack easily integrates with a bunch of OSS stack used by cloud native companies.

hnarn8• 6 years ago

Some people are probably going to throw some shade on me for saying this since it's so out of fashion but in my mind, when it comes to some types of basic monitoring (SNMP monitoring of switches/linux servers, disk space usage, backups running and handling them when they don't) then Nagios does get the job done. It's definitely olives and not candy[1] but it's stable, modular, relatively easy to configure (when you get it) and it just keeps chugging away.

If anyone has any Nagios questions, I'd be happy to answer them. I'm a Nagios masochist.

(Also, I can recommend Nagflux[2] to export performance data, metrics, to InfluxDB because noone should have to touch RRD files)

[1] https://lambdaisland.com/blog/2019-08-07-advice-to-younger-s...

[2] https://github.com/Griesbacher/nagflux

brightball• 6 years ago

Ever looked at Zabbix?

hnarn8• 6 years ago

I haven't, unfortunately, but it looks promising from just looking into it briefly. Open source monitoring is always an area that needs more competition.

thethethethe• 6 years ago

Possibly stupid question:

Can you use Nagios to stream metrics exported from your applications binary in real time?

For example, can you use Nagios record each http request processed by your application webserver, tagged with http method, code, latency etc?

hnarn8• 6 years ago

Nagios is not so much a "recorder" as it is a "state inspector", basically plugins run on a schedule and inspect that things are up to spec. The situation you describe may be better suited for something like the ELK stack which can hook into your HTTPD logs.

BurritoAlPastor2• 6 years ago

Nagios is the Jenkins of monitoring. It's popular because you can get it running in an afternoon, and it's easy to configure by hand.

It then rots within your infrastructure, because it resists being configured any way _except_ by hand. I've built two systems for configuration-management of Nagios (at different companies), and it's an unpleasant problem to solve.

Prometheus's metric format and query syntax are cool, but the real star of the design is simply this: you don't have to restart it, or even change files on your Prometheus server, when you add or remove servers from your environment.

hnarn8• 6 years ago

Yeah, I was in no way insinuating that Nagios is superior in general, or even to Prometheus, just that it does the job well for some use cases. Monitoring is tricky and you definitely need a tool box because each problem has a different optimal solution.

znpy4• 6 years ago

I have to use an icinga instance from time to time (icinga is a nagios fork). I really can't see the value, beyond seeing if a service is up or down.

I'm surprised no one has named Zabbix. Zabbix is way better. I hadn't the chance to use Zabbix past 4.something but it's worth it.

I've been using Prometheus/grafana and frankly the value I see is it's out of the box adaptability at capturing a mutating data source (example: metrics about ephemeral pods Una kubernetes cluster).

mekster6• 6 years ago

you can get it running in an afternoon, and it's easy to configure by hand.

This read like a joke. Nagios looks like it's from stone age having files in cgi-bin folder with unnecessary complication to installation and management, unless they made it any better at some point.

egwor• 6 years ago

We run this at work, and I have (to put it politely) severe reservations. What does these functions for you:

- Realtime GUI which works with windows 10 (we have a web site and nagastamon) - aggregation of alerts / alert roll up - sharing filters - summary + description - temporary downtime of alerting - message rate suppression (to stop floods) - filtering of columns/ordering etc. - bulk actions for closing alerts

I've come from using: - HPOV (great but can't handle bursts of alerts) - email (everyone has to have filters, can't handle bursts of alerts that well (runs everyone out of email space!), prone to failure/delays due to email - home grown solution

hnarn8• 6 years ago

I'm not sure what you mean by "this" but I suspect you mean Nagios XI or some other corporate offering. I was referring to the core monitoring component and its forks/derivatives.

Other than that I don't quite understand the point of your comment, you say you have "severe reservations" but many of the points you list are available even with Nagios core, and most of them are available in other Nagios variants.

9thbyte• 6 years ago

This seems like a lot of work when you could use Zabbix.

¯\_(ツ)_/¯

linsomniac2• 6 years ago

I've been doing monitoring of our ~120ish machines for 3-4 years now using Influx+Telegraf+Grafana, and have been really happy with it. Prior to that we were using collectd+graphite and with 1 minute stats it was adding some double-digits %age utilization on our infrastructure (I don't remember exactly how much, but I want to say 30% CPU+disk).

Influxdb has been a real workhorse. We suffered through some of their early issues, but since then it's been extremely solid. It just runs, is space efficient, and very robust.

We almost went with Prometheus instead of Influx (as I said, early growing pains), but I had just struggled through managing a central inventory and hated it, so I really wanted a push rather than pull architecture. But from my limited playing with it, Prometheus seemed solid.

drez• 6 years ago

I just wanted to echo this sentiment. Influx had its fair share of issues a few years ago (v0.8 migration, changing storage engines, tag cardinality issues), but the latest v1.x releases have been solid. I have been using the TIK stack (I use Grafana instead of Chronograf) for monitoring several dozen production-facing machines for 2 years now without a single issue, which I would very much count as a win.

I just hope they learned their lesson for the v2.0 release...

rehevkor52• 6 years ago

It's so much easier to write incorrect/misleading queries in influxql than in promql. And you can't perform operations between two different series names in influxdb, last I looked. That makes it impossible to do things like ratios or percentages unless you have control over your metrics, and structure them the way influx likes. Also, no support for calculating percentiles from Prometheus histogram buckets.

mhall11920• 6 years ago

You can do these things (and much, much more) with the new Flux language

valyala12• 6 years ago

I really wanted a push rather than pull architecture

Then try VictoriaMetrics - it supports both pull and push (including Influx line protocol) [1], it works out of the box and it requires lower amounts of CPU and RAM when working with big number of time series (aka high cardinality) [2].

[1] https://victoriametrics.github.io/#how-to-import-time-series...

[2] https://medium.com/@valyala/insert-benchmarks-with-inch-infl...

aequitas• 6 years ago

For those who still remember Graphite, the team over at Grafana labs have started maintaining Graphite-web and Carbon since 2017 and it is still in active development getting improvements and feature updates. It might not scale as well as any of the other solutions, but for medium size or homelab setups it's still a nice solution if you don't like PromQL or InfluxQL.

https://grafana.com/oss/graphite/

lma212• 6 years ago

for simple and fast monitoring solutions, i always opted for collected + graphite + grafana. In a containerized environment, it's so easy to deploy (0 configuration by default) and monitor a set of 50-100 nodes. Beyond that, disk tuning and downsampling (pre-aggregation) rules become important.

rehevkor52• 6 years ago

If you're still on graphite and need scaling, metrictank is something to consider.

Aaronstotle• 6 years ago

I recognized the name almost immediately, cool to see a blog post from a member of the monero community.

crecker3• 6 years ago

haha thanks!

mickeyben• 6 years ago

We're currently using InfluxDB but maintenance is something we'd like to stop doing on our monitoring stack.

Datadog is too expensive because of the number of hosts we have. So we're thinking of eventually going to a hosted InfluxDB setup.

But we also want to revisit other hosted solutions. Does someone have some experience with using Cloudwatch + Grafana? I've used Cloudwatch many years ago and it was clearly subpar to something like Influx. Is is better nowadays?

metalliqaz• 6 years ago

I use Telegraf, InfluxDB, and Grafana for my home network, in addition to monitoring my weather station data. It works incredibly well and is so simple compared to the "old" suite of tools such as RRD and Cacti.

sahoo• 6 years ago

How does this compare to grafana, Prometheus and node monitor

jakobdabo• 6 years ago

I still can't find any alternative to the old, RRD-based Munin. It is so simple. You want to add a new server to monitor? Just install there the node part, enable any required additional plugins (just by creating a couple of soft-links), add one-line configuration to the main server with the new node's IP address, and you are done.

Also, the aesthetics of the UX, you see all the graphs in one single page[1], no additional clicks are required - a quick glance with a slow scroll and you can see if there were any unusual things during the last day/week.

[1] - publicly available example, found by googling - https://ansible.fr/munin/agate/agate/index.html

mhall11920• 6 years ago

That's more steps than using InfluxDB and Telegraf.

Check out: https://github.com/influxdata/community-templates/tree/maste...

ecoqba11• 6 years ago

We switched from InfluxDB to TimescaleDB for our IoT solutions. InfluxDB is very difficult to work with large datasets and enterprise/region compliance. We ingest around 100MB data per day and growing.

mhall11920• 6 years ago

That doesn't actually sound like a large dataset. Can you describe what kind of problems you faced with InfluxDB?

valyala12• 6 years ago

100MB data per day looks like a tiny number. Our users ingest 100GB data per day at the ingestion rate of 1M data points per second on a single-node setup [1].

[1] https://victoriametrics.github.io/CaseStudies.html

DevKoala• 6 years ago

I’d replace InfluxDB with Prometheus. InfluxDB has been a PITA for me.

mavam2• 6 years ago

Does anyone have experience with an InfluxDB-based netdata setup?

When we evaluated multiple options, we found that netdata does a lot of the heavy lifting. What are the advantages of a CollectD-based setup?

zacksh• 6 years ago

Well, that's the beauty of Netdata, no need to configure collectors. As for InfluxDB integration, it's fairly easy to export from Netdata using what InfluxDB calls the line protocol format (the "plaintext interface").

https://learn.netdata.cloud/docs/agent/exporting

Monitoring your own infrastructure using Grafana, InfluxDB, and CollectD

Related Stories