InfluxDB v0.10 GA with hundreds of thousands writes/sec, 98% better compression

https://influxdata.com/blog/announcing-influxdb-v0-10-100000s-writes-per-second-better-compression/

by pauldix • 10 years ago

152 75 10 years ago

influxdata.com

illumen• 10 years ago

But does it work? It wins the most buggy database servers I've ever used in 20 years award.

Is writing less than 20MB/sec of data something to brag about?

sytse• 10 years ago

We're using it with GitLab.com and we like it. 0.9 didn't work at all due to the volume but with 0.10 everything is functioning OK

fweespeech3• 10 years ago

A full cluster or a single node?

pauldix14• 10 years ago

Am I correct in assuming that you got to the 20MB/sec number by taking our 3 bytes per point number times the number of data points?

The input is actually much higher than that. Data points over the network look like this:

cpu_idle,host=serverA,region=uswest value=23.0 1454617920

That's actually a toy example. Most real data would probably have more tags and longer measurement names. Obviously that's much more than 3 bytes.

We persist that to disk in a write ahead log (WAL) and then later we can do compression and compactions on the data to squeeze it down to 3 bytes per point. However, that takes more than a single write against the disk to get to.

Run a load test against it. See how much network bandwidth you can use. See what your HD utilization looks like. My guess is you'll be surprised by what you see.

switch007• 10 years ago

And about the bugs...? My experience with 0.8 and 0.9 was somewhat sub-par. I personally am awaiting 1.x and it wouldn't surprise me if others were too.

SkyRocknRoll• 10 years ago

We are using 0.10 GA in production. We have seen lot of disk space reduction from 22GB to 700MB and performance improvement order of magnitude. Influxdb is the modern timeseries database.

Keep Rocking !!

Shish2k• 10 years ago

I wonder what sort of data you're storing with that reduction? I'm using it as a drop-in replacement for Graphite (with Grafana as a front-end, loving both BTW) and just had disk space reduced from 140GB to 250MB :D

ddorian432• 10 years ago

They did the mongodb approach. Build a shitty db and then build a normal one and claim 100X faster.

And then add synchronized disk commits (like postgresql has always done) and performance goes puff (meaning lower than pg) (example: elasticsearch, mongodb).

pauldix14• 10 years ago

fsyncs happen on every write. They're durable. This release may be 100x faster than our previous release, but it's also a competitor with other solutions.

We haven't run any comparisons vs. other solutions, but we will do that soon and I expect that we'll be competitive and better in some cases.

CyrusL2• 10 years ago

Very tasteful way to handle the parent comment :)

Does Todd Persen talk about durability testing in the video or is more about performance benchmarking? I haven't watched yet.

https://vimeo.com/153802549

fapjacks• 10 years ago

This is not the first time I have seen an anti-InfluxDB post that also subtly pushes PostgreSQL in its place. Now I totally agree and I am not happy with InfluxDB's performance, and maybe I'm just being paranoid, but it's interesting I've seen this a handful of times.

flowless2• 10 years ago

I would like to see actual comparison of both. I'm going pg way myself but due to millisecond precision of timestamps required for handling scientific data.

pauldix14• 10 years ago

Have a look at this release. The write performance is massively better than it was. We did tests writing up to 100B points against a single server.

The next release will focus on improving query performance, but this one still works for many cases.

ddorian432• 10 years ago

Nothing against influxdb. But if you have 100X, you probably were doing something shitty before.

I use postgresql as comparison because antirez did in http://oldblog.antirez.com/post/redis-persistence-demystifie... (and ~none else has done something similar).

zenlikethat• 10 years ago

I have been forwarding Riemann metrics to it and visualizing with Grafana, pretty fun so far. Love that it's written in Go and fairly lightweight on resources compared to JVM-based technologies.

bfrog• 10 years ago

Been using 0.9 for a few months now and been pretty happy with it. Though I'm not really looking forward to figuring out how to move from telegraf 0.2 to 0.10, since that had some breaking changes.

I do love the simplicity of influx+telegraf, kapacitor also looks cool. Chronograf seems like a bad version of grafana still... maybe in the future if its FOSS and somehow manages to be better than grafana I'd use it.

kev0096• 10 years ago

Be _very_ skeptic of this, I wasted weeks trying to get git master versions of this not fall over including tsm1 storage engine. They are very good at marketing and made nice query and API entry, but data storage has been a total shit show.

Looking at less than a month of issues:

* https://github.com/influxdata/influxdb/issues/5440

* https://github.com/influxdata/influxdb/issues/5482

* https://github.com/influxdata/influxdb/issues/5534

* https://github.com/influxdata/influxdb/issues/5540

If you need something that won't fall over, it's not glamorous and a bear to setup, but OpenTSDB will sail with massive load once you've gotten it running.

pauldix14• 10 years ago

Not sure what issues you're seeing, but we have people running this in production at significant scale.

One of the issues you linked to was for 0.9.6. Others were there because they had super high tag cardinality and not enough memory to actually run.

When people post comments like "I have 50 million series and it crashes on my box with 2GB of memory!!!", they're not relevant. You shouldn't expect miracles...

cantbackup• 10 years ago

How about issues like these? https://github.com/influxdata/influxdb/issues?utf8=&q=is%3Ao...

In my case I had a 0.9.4 influxdb setup fed by statsd for about a week. The server hard crashed the first time I tried to back it up.

kev0096• 10 years ago

No, and you're countering rhetoric with rhetoric so here are the specs: bare metal 96G RAM, 16 cores, 36 disks (also tried 36 SSDs). Nothing exotic, and something similar you should be using to test before releasing another ill-fated storage engine based on "It works on my Macbook Pro with an artificial load test, ship it!". On the same box carbon is usable, and TSDB can scale linearly across a cluster of these machines. Influx burns down as soon as more than a few GB of metrics are persisted even after backing off collection.

CSDude• 10 years ago

I just love the ability to GROUP BY queries, they work so fast and saves me from writing a stupid map-reduce style job for that basic thing. I am building a time series nice analysis product with it and this change would work very fine.

thecourier• 10 years ago

can you efficiently store logs in InfluxDB? would love to read if somebody have tried that before... (just wondering)

lobster_johnson• 10 years ago

InfluxDB is optimized for storing samples. I don't know for sure about 0.9+, but I can't imagine the architecture has improved to the point where it would be happy about storing large amounts of text data.

pstuart• 10 years ago

Perhaps if the logs used logfmt with a parser as an intermediary.

piran• 10 years ago

We store some logs at a small scale (5k/s) and queries are fine 0.9.5. You just have to be smart how you do tags still. Hopefully 0.10 is the same.

effenponderousa• 10 years ago

I have nothing to do with influx and I probably won't in the future.

There are way too many haters in HN. You venomous minority who shit on every bit of good news that isn't yours -- keep your negativity to yourself.

You fucking monkeys infected with rage.

donatj• 10 years ago

On the one hand I agree with your sentiments, on the other hand I disagree with your approach.

la64702• 10 years ago

Looks like all the hordes of oldies from slashdot is now coming to HN

dang• 10 years ago

It's true that there's too much negativity on HN, but it's also true that people contribute to it without meaning to. Your comment kind of demonstrates that.

On the other hand, your use of the word 'minority' there was both astute and thoughtful.

la64702• 10 years ago

I don't like the fact that old data will continue to use the old engine after the upgrade.

jmcgough• 10 years ago

There's a migration script that can be used for moving old data to the tsm engine.

XorNot• 10 years ago

Is the text ingest engine usable with TCP yet? UDP is a bit weird for this when I may care greatly about whether my server is successfully feeding a stream of lines too influxdb.

pauldix14• 10 years ago

We don't have the TCP listener wired up yet, but it's on the roadmap. Thanks for bringing it up, I'll try to get it prioritized for sometime in the next few months.

kyloon2• 10 years ago

I had another go at this new release after failing to get any of the 0.9.x releases to suit my use case of massively high volume writes. Now that I managed to get some of my data in it, aggregation queries seem to be lagging in speed, I would like to know more about what is actually being done currently to improve the query speed rather than waiting out for the v0.11 release as I am deciding whether InfluxDB is the way forward for my use case.

pauldix14• 10 years ago

The new query engine work should get merged into master early next week. You'll be able to test against a nightly build then to see what kind of performance improvement will be part of 0.11.

What kinds of queries are you seeing poor performance with? That would help us troubleshoot and improve, thanks.

kyloon2• 10 years ago

Thanks for the update. Most of the queries I am testing now are 'mean' and 'group by' aggregation for downsampling and visualization in Grafana. Looks like the problem is CPU bound as I see all 8 cores got maxed out when processing the queries. Currently there are about 50 series and 5B data points in total.

stock_toaster• 10 years ago

I would love to see some good (and up to date) documentation on replacing graphite with influxdb (retention, rollups, etc).

Last time I tried it out I vaguely recall that configuring rollups was kind of painful -- lots of nearly duplicate CS queries, even for a relatively small number of series.

eklavya• 10 years ago

A lot of comments have mentioned Time Series use. How does it compare to Cassandra?

chaotic-good2• 10 years ago

It depends greatly on the shape of your data, but with regularly spaced timestamps at second level precision and float64s, we’ve seen compression reduce each point down to around 2.2 bytes per point.

AFAIK you're using float64 compression scheme from "Gorilla" paper. 2.2 bytes per data point is possible with it but only on data that doesn't utilize full double precision (example: small integers converted to float64). You should compare compression algorithm used by TSM storage engine with zlib or any other general purpose compression algorithm, otherwise this number will be meaningless.

Dylan16807• 10 years ago

Honestly if you have a series of increasing unix timestamps you should be able to do much better than 2.2 bytes each.

chaotic-good2• 10 years ago

Even simple delta+RLE can squeeze increasing unix timestamps up to few bytes. I'm talking about floating point data compression. Even the specialized state of the art algorithms can't guarantee you 2.2 bytes per float in general.

AYBABTME• 10 years ago

I don't get all the hate. Reading the comments here yesterday and then today again, made me really question what's going on in this community.

They (InfluxDB) made huge progress, they work hard on an open source DB (and ecosystem) and people insult them and question Paul Dix's >30 minute response time to your support questions in a HN thread.

InfluxDB v0.10 GA with hundreds of thousands writes/sec, 98% better compression

Related Stories