Timeseries data storage in MongoDB

http://www.slideshare.net/sky_jackson/time-series-data-storage-in-mongodb

by seigenblues • 15 years ago

39 25 15 years ago

slideshare.net

yannis3• 15 years ago

Interesting presentation, would do better with some more details in a blog or pdf.

seigenblues5• 15 years ago

thanks, i'll try and write them up. any particular areas you'd like to see expanded?

yannis3• 15 years ago

I am particularly interested in the software implementation. I am a Mechanical Engineer, involved with high rise construction. Looking to disrupt Building Management Systems.

cstuder• 15 years ago

Being in a similar industry, I share the sentiments of the last couple of slides: Dataloggers are expensive and horrible pieces of hardware. Proprietary solutions with no regard for real-life scenarios (Limited connectivity, power failures, connection failures, weird and inflexible data formats...)

I would love to have a look at their Arduino based solution.

iskander3• 15 years ago

Are there advantages over storing data in HDF? I've been working with a few hundred gigabytes of financial data this summer and I'm finding that python's data-oriented libraries (h5py, numpy, scipy, matplotlib, scikits.learn) cover my needs.

seigenblues5• 15 years ago

depends on your usage. The rest of the toolchain is all python, numpy, scipy, matplotlib especially.

This might be poorly titled: it's not just about the storage as it is about the aggregation of disparate sensor data into coherent, continuous data streams.

iskander3• 15 years ago

I'm totally ignorant of mongodb: what does it do for you (in the way of data aggregation) that's not easy in numpy?

moe• 15 years ago

Aw, this was physically painful to skim.

What you really want for time-series data is a column db such as cassandra (or vertica etc.), perhaps HBase, perhaps a RDBMS, or perhaps a plain old log-file.

What you most definitely don't want is Microsoft Access or MongoDB. Thinking about it, MS Access might still work to a degree.

seigenblues5• 15 years ago

glad you didn't like it.

Vertica was not free, HBase was really heavyweight, & I knew i didn't want RDBMS. The sensor data itself is plain csvs ;)

Cassandra still does look interesting, although it looked like it would take me much longer to get it going; perhaps it will get rewritten. (The mongo solution only took a week to get something usable.)

rbranson4• 15 years ago

Curious to hear why you thought it would take longer to get going with Cassandra. A single-node Cassandra cluster can be up with a 1-line command. Was it the availability of clients or interface abstractions? I understand that tutorial & documentation is not as readily available, so that's understandable.

rbranson4• 15 years ago

As long as it all the data for all metrics fits in RAM, it will work great in MongoDB. Of course, as soon as it doesn't, you're totally hosed.

To expand on the parent, a column store like HBase or Cassandra is perfect as each row can represent a timeslice and then each column can represent a single event or record within that timeslice. As the row gets evicted from it's initial storage in memory, it will be written to disk contiguously, in sorted order, and batch reads of this data is sequential.

It is possible to use MongoDB to store a timeslice as a document, but it is not designed to scale out to store very large numbers of columns within a single document.

seigenblues5• 15 years ago

happily, this is not a synchronous application with tens of thousands of concurrent users, so "totally hosed" for us may have a very different definition.

What do you mean by "very large numbers of columns" ? I've seen some mongo users with very rich, i.e. large, document models.

amalag2• 15 years ago

MongoDB needs indexes to fit in memory, not the entire dataset. This is an important distinction.

hogu2• 15 years ago

Is it just about the number of dimensions? Seems like you'd get decent performance with mongodb if each timeseries is a collection, and each entry was another document?

ericHosick• 15 years ago

For a temporal or time based key value store (I think this is kinda what the presentation shows) I used a collection that was something like:

Temporal Collection { _id: "X1", data_temporal : [ { time_start: SomeDate, time_stop: SomeDate, _id: "ID2" }, { time_start: SomeDate2, time_stop: SomeDate2, _id: "ID2" }]

Data Collection { _id: "ID1", parent: X1, data: { field1: "some info", field2: 34 }, _id: "ID2", parent: X1, data: { field1: "Some info new", field2: 34 } }

What is cool about this is that if you have access to the data like ID1, you can easily find out when it was added and how it changed.

If you have access to the temporal ID, X1, then at any time you can see what the data looked like.

If you need to relate data, the "foreign key" used is the data_temporal ID. In this way, it is possible to ask what your key value store data looked like at any time.

But, this could be off from the article.

This also works quite well in a relational database.

luigi• 15 years ago

I saw this presentation live at MongoDC and it was awesome.

snowwindwaves• 15 years ago

he needed to find this board for his datalogger http://www.amazon.com/Webcontrol-Universal-Temperature-Humid...

amalag2• 15 years ago

What about Hadoop? You can also use Hadoop as a backend for hadoop style filesystems. What about using Hive? Does it also require a fixed schema?

Timeseries data storage in MongoDB

Related Stories