Ask HN: Best time series database currently

https://news.ycombinator.com/item?id=15816596

by mads • 9 years ago

21 16 9 years ago

news.ycombinator.com

I am looking for a database to store time series data from sensors and found CrateDB. It looks very interesting. Does anyone have experience with it?

I need something that will scale well horizontally, which it seems CrateDB should be able to do, and CrateDB also has a MQTT broker built into their enterprise version.

Or maybe some other alternative that will scale well?

I am looking at influxdb and Prometheus also but the MQTT broker in CrateDB really appeals to me.

kevindqc• 9 years ago

I don't know anything about this, but it might help if you provide information on what you want to do with the data.

mads3• 9 years ago

My main concern for now is to be able to store/write the data collected from a million IoT devices in a scalable fashion, so there is not really a requirement for what to do with it other than, when the data is there it will be analyzed on an adhoc basis and then we will see what we can do with it.

brianwawok• 9 years ago

That is going to be hard to achieve.

To know how to store the data, you need to know how you will query it.

If you know how you will query it, you can say devise a way to store it in Cassandra.. which will scale up to PB.

If you just throw in a PB of data, I am not aware of ANY system that is going to let you drop ad-hoc queries at the data and get fast answers. You effectively need to load most / much of the data off disk to process it.

If you only want to store it and not query it yet.. store it off on flatfile. When you decide how you will query it, load it from the flatfile and switch writes to your new system.

ehllo3• 9 years ago

Just have a look at Timescale. It's a open source postgresql extension, and can sit side by side with other stuff. Scaling postgresql should be no problem for you.

https://www.timescale.com/

mads3• 9 years ago

Cool thanks. I will check it out.

jurgenwerk• 9 years ago

I don't think PostgreSQL scales nice horizontally. I do use TimescaleDB though and it scales really nice vertically, but would probably be hard to make it run on multiple machines.

ehllo3• 9 years ago

I see timescale more as an alternative option, since the OP also aksed for this. - And you're right, since Timescale is an postgresql extensions, it also has the same benefits/problems as postgresql itself in this manner.

johann83843• 9 years ago

I haven't gotten to really use it that much, but I'm a fan of this tool!

IpV8• 9 years ago

It may be worth looking into what AWS has to offer. Their IoT offerings seem to be pretty solid.

dyeje• 9 years ago

We use DynamoDB at my place. We had to make some tooling around it and figure out some growing pains, but now we have an extremely scalable solution.

Maven911• 9 years ago

how about kdb, the time series in memory db, long history in financial services

johann83843• 9 years ago

OpenTSDB scales well, 10 million DPS writes on a 36 node cluster for example. It's not easy to get working well at scale out of the box. That is something we really need to work on in that community. There is a project called Splicer (github.com/turn/splicer) that will shard the incoming queries into 1 hour blocks and cache the 1 hour blocks, it also sends the queries to the same node where the region server is that hosts the data. This makes the queries VERY fast.

Ask HN: Best time series database currently

Related Stories