TileDB: Storing massive dense and sparse multi-dimensional array data

http://www.tiledb.io/

by rajnathani • 9 years ago

180 72 9 years ago

ttd• 9 years ago

Looks interesting... Here's the publication that it appears to have formed from: https://dl.acm.org/citation.cfm?id=3025117 (VLDB 2016).

styx31• 9 years ago

Also available directly here: [PDF, VLDB 2017] https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...

dx0342• 9 years ago

I would like to see some information on compression ratios. Compression can often significantly improve performance (less data to load/save, more in memory or cache) and esp dense arrays should compress easily. They mention that compression is supported but I'm not sure if that's done internally or just applied to the files they create.

jbooth• 9 years ago

Floating points compress poorly, though.

dogruck• 9 years ago

Not always poorly -- especially if you preprocess by calculating running diffs of consecutive values.

wenc2• 9 years ago

Not always [1].

Also in many time-series applications involving sensors whose readings don't fluctuate that much, process historians often apply deadband compression (i.e. store only one value if it is within a certain band). The type of compression is lossy and sometimes a bit controversial for high-fidelity uses, but often results in efficient storage.

[1] zfp, fpzip: https://computation.llnl.gov/projects/floating-point-compres...

srean4• 9 years ago

There have been a few responses along the lines of 'not always' but what you say is indeed largely true.

There is another thing that is worth considering and that is the algorithms (and even the theory) that works well for compression of discrete sources are not well suited for compressing real numbers (floating point numbers aren't, but, they are the poor man's reals). On the theory side, this bothered Claude Shannon enough that he decided to revisit this later in his career to create rate distortion theory, he knew that there was some unfinished business in information theory.

We do have sort of a chicken and egg problem here, especially when we want store a lot floating points for a ML workload. Learning how to compress and learning the underlying distribution are equivalent problems. If we have already learned the model, then yes we could compress the data well. But when we haven't, then by definition we wouldn't have the knowledge to do a good job of storing the data in a well compressed form. After we have acquired the knowledge to compress well, we don't really need the compressed data anymore to learn the model, we already have it. One way to address this would be to do both incrementally and simultaneously.

jakebol9• 9 years ago

Jake from TileDB, Inc. wenc and srean are right. techniques such as those used in zfp and fpzip wenc mentioned are also used to compress real world las file (point cloud) datasets. For the moment we are only focused on lossless compression (scientists are paranoid about losing data), but there is definitely room to explore integration with lossy compression as well. Machine learning applications often do not need full precision so intelligent forms of lossy compression are useful.

Another cool research application of TileDB that extends the storage library with the VP9 codec can be seen here: https://homes.cs.washington.edu/~magda/papers/haynes-sigmod1...

JFFalcon• 9 years ago

The Learn More section has a very nice description of the concepts. The compression page [0] seems to say that they group each attribute for a fragment together and then apply compression.

[0] https://docs.tiledb.io/v1.0/docs/compression

davedx• 9 years ago

Sounds ideal for storing training/intermediate data for machine learning. Niche competitor for Cassandra in this space?

speedplane2• 9 years ago

Most models I've seen have at most a few hundred MB of parameters, have complex connections that can only be modeled as a set of many different 1 or 2D arrays with different sizes. Further, this DB stresses that it handles sparse sparse data, any most ML data is not sparse.

It doesn't seem like the best application. On one of their pages it mentions they ingest BAM records, which is for biological sequences. I'm guessing some DNA storage applications.

wenc2• 9 years ago

most ML data is not sparse.

This brings up a question: in what fields does one find heavy use of large, sparse matrices that need to be persisted and queryable?

In my mind, sparse matrices typically occur in the context of graphs/relationships, e.g. PageRank, logistic networks, adjacency matrices, etc. They also tend to be a property of Hessian matrices (2nd order derivative for a multivariate system). But typically these are intermediate quantities that are discarded after a computation completes.

srean4• 9 years ago

most ML data is not sparse.

As an ML guy I can say that sparse is pretty common in ML. Text data, market basket data, graph Laplacians, adjacency matrix, sequence fragment data are all rich in sparse matrix computation and storage operations. For a moment I thought the "not" was a typo. Lack of support for sparse matrices often becomes a serious inconvenience in a tool, very happy that TileDB folks have given thought to the sparsity requirement from get-go

marco-lavag• 9 years ago

I wish they had python bindings.

castis• 9 years ago

Their Python, R, Matlab, and Excel bindings are in-progress.

Kiro• 9 years ago

Would it make sense to store a tilemap for a game in this? Normally I use NoSQL for that and just store the tilemap as a JSON array. I need to be able to update and retrieve single cells easily (this is for a backend in a multiplayer game).

k__• 9 years ago

I don't know, tiledb seems rather low-level and many databases have geospatial indexing.

mrdrozdov• 9 years ago

Could I use this for deep learning research? It’s common for my models to be anywhere from 100mb to a few gb. Although, I’d find it more useful for reading batched training data.

stavrospap9• 9 years ago

Stavros from TileDB, Inc. here: TileDB could be useful for storing your training data (in some storage backend of your choice) as well as your intermediate data (as davedx pointed out). But this would make sense only if your data are truly large and cannot fit in main memory. In fact, we are looking forward to seamlessly integrating with systems like TensorFlow, but we would rather wait until we can bring some value to applications with very large storage requirements.

rajnathani• 9 years ago

The underlying technology [1] for TileDB came up with MIT and Intel working together.

And TileDB, the company, formed out of it, has recently received funding [2] from Intel in Intel's latest tranche of investments.

[1] https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...

[2] https://techcrunch.com/2017/10/19/data-is-the-name-of-the-ga...

grabcocque• 9 years ago

Good to know private shareholders are getting even richer off all that taxpayer research money that's pumped into MIT.

mac01021• 9 years ago

Given that this is an open-source project, Intel's involvement doesn't seem too sinister to me.

stavrospap9• 9 years ago

Stavros from TileDB, Inc. here: This comment gives us the opportunity to explain a bit the history of the company and our thesis. TileDB was my research project while I was working as a full-time researcher at Intel Labs and was stationed at MIT. This was part of a collaboration between Intel Labs and MIT, called Intel Science and Technology Center (ISTC) for Big Data. Intel funded MIT (and several other universities) for 5 years, while mandating that any tech produced in the center should be open-source with a non-restrictive license (like our MIT license), and without any strings attached. So ISTC was for pure research. We created TileDB, Inc. so that we can continue working on TileDB beyond the graceful termination of the ISTC, since we believed that the tech (and future vision) can significantly contribute to the scientific community (there was already some early evidence through our work with the Broad Institute on genomics). The company will continue to respect and contribute to the open-source community, while trying to bring together very talented people to transition the tech from a research project to a production-ready software.

_pmf_2• 9 years ago

Python, R, Matlab and Excel

Mentioning Matlab and Excel immediately puts the product in the category "they know what they are doing" as opposed to "another group of sophomores trying to reinvent data science".

I'm still waiting for a raw data dump with "avoid copies at all cost" access for very raw, very verbose vehicle and manufacturing data that is in 99.9 per cent never accessed, but must be analyzed when errors are detected late in the process. I.e. there's practically no transformation to apply during storage, but upon access, transformation must be done.

If TileDB is kind of like a more structured, more low-level struct oriented Redis, it is a very welcome addition.

srean4• 9 years ago

Apart from a diminishing population of old farts and a clutch of national labs and universities that Mathworks keeps well greased, is MATLAB even relevant anymore ?

_pmf_2• 9 years ago

Pervasive in the German automotive industry.

jakebol9• 9 years ago

Jake from TileDB, Inc. Engineering as a discipline is conservative (for good reason). The tools and the processes they use change slowly. Matlab is still entirely relevant both in the sciences and in industry. There are people with huge amounts of domain knowledge (who may only know Matlab or Excel) that are increasingly called upon to analyze and interpret larger amounts of data. These "old farts" are the people engineering, designing, and debugging our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. There is tremendous value in building bridges across communities and across generations here.

I say this as someone who helped in small ways to develop open source alternatives to Matlab.

jeff_science• 9 years ago

Yeah, those deep-pocketed government lab and university researchers are clearly the dominant revenue source for MathWorks. Everybody in the corporate world is super nimble and moved on to Numpy as soon as they could.

bigger_cheese• 9 years ago

It is used pretty extensively in my industry (industrial manufacturing). There has been some talk about R but not sure it will be widely adopted.

You are right that engineers are conservative which no doubt plays some role. But I'd say technical debt and legacy code plays much more of a role.

Most Engineers (Chem, Mech, Matls etc) are not exposed to code during university. Maybe it is changing now but it is slow process. Often first exposure comes when the engineer enters industry and is asked to work on existing model - usually under supervision of a senior engineer. You learn whatever the language the senior engineer knows and that is typically what they learnt from similar mentorship - it is often Fortran or C/C++. Our Industrial Process has not changed dramatically in last 30 years. So once efficiently written and accurate simulation code exists very little reason to rewrite (if given the choice to rewrite an existing model in whatever the current language du jour is or continue to hack on something that already exists most engineers - at least the ones I know, would chose to hack on existing code). For pretty much this reason there is a heap of Fortan that is still alive in my org (with roots that can be traced back to the '80's). I think it's probably worth mentioning a lot of engineers don't approach code thinking about algorithms - it's equations i.e. I need to write something to solve Bernoulli's equation - or Ergun's equation or similar. Linear programming (i.e solving simultaneous equations) is the other main reason to write code and hey MATLAB does this pretty well...

The other reason for being "gunshy" about new technology at least in my org is that a lot of the senior engineering people still remember when we got bitten by investing in Microsoft "Stack" in the late 90's. There were a lot of Modelling done with VB6 and Access which tuned out to be a technology dead end. Access databases in particular have plagued our org in many cases multiyear effort to migrate something out of Access. Open source probably protects against this but everyone wants to be sure any new technology will still be around in 15 years time. I think that is why people in my org are finally starting to look at R it has passed that initial hurdle and there is some confidence it is not just a fad...

StanSeltser5• 9 years ago

Stanislav Seltser, Petacube _pmf: think of distributed persistent numpy arrays as opposed to key-value pairs in Redis. The idea you will not pull data to do computation, you will push your code in. pretty good for archiving due to compression

brootstrap• 9 years ago

anyone have experience using one of these array data stores to handle large amounts of weather forecast data? At my job we've come up with our own clever postgres solution to handle the large amounts of gridded binary data. Essentially we are inserting large 3D arrays (a series of images where each pixel represents a geographic location and a value, like windspeed or temperature or rainfall) into our postgres. Our solution is solid but am always on the lookup for new novel approaches.

jakebol9• 9 years ago

Jake from TileDB, Inc. I think this would be an ideal workload for array data stores (NetCDF a standard in this area uses HDF5 under the hood). You have <N> number of attributes that you want per grid-point over time (and you want to append to the time dimension). If you are ingesting Grib2 files then you can take advantage of compression as well. An array data store like TileDB should offer advantages for fast access, as you can get a pointer directly to the stored array and do not have to access the (serialized) data over a socket, especially if you are only interested in a subarray of the dataset.

oliviervg1• 9 years ago

Hi both, this is exactly something that I’m looking at doing. We’ve got about 10TB of netcdf data coming in everyday and we’re looking for a cost efficient data store to provide fast access to individual grid points. S3 has proven to be too slow.

Any chance I could pick your brains about using either Postgres or TileDB?

Thanks!

StanSeltser5• 9 years ago

Stanislav Seltser, Petacube Inc

ingesting data into postgres makes sense for sparce data but not for dense data because it is waisting a lot of of space due to storage of coordinates with every data point and every weather variable. If you are using NOOA grib2 forecast files those are dense . Not to mention losing compression in postgres. TileDB will store data compressed, the dimension coordinates themselves will be compressed, plus column storage (each NetCDF variable) will make retrieval of dense weather data blazingly fast as oppose to postgres where you will have to scan the whole table

ankitrohatgi• 9 years ago

I am curious about how the query performance compares to working with JSON files in Spark for ~100GB data.

maxpert2• 9 years ago

They have some benchmarks in paper https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...

jakebol9• 9 years ago

Jake from TileDB, Inc. here: Depending on the structure of the JSON files you are querying you maybe able to take advantage of columnar compression and massively reduce the dataset size (especially if the json files contain numeric data). Also, repeat queries will not have to re-parse the JSON files. This may speed up queries quite a lot, but it depends on the specifics of your problem.

StanSeltser5• 9 years ago

Stanislav Seltser, Petacube you are talking comparing structured workload(array-based TileDB) to unstructured one (JSON+Spark). Once you convert your JSON to sparce array structure (one time conversion) TileDB will beat Spark+JSON by several orders of magnitude. Caveat: assuming your spark+json workoad is a some heayy processing not a lightweight one.

maxpert2• 9 years ago

Looks nice, still have to read the whole paper. Seems like it's most useful for sparse arrays. Maybe we will get a golang port :P

kuwze• 9 years ago

I wonder how it compares to SciDB[0] which is also used for storing multidimensional data.

[0] https://en.wikipedia.org/wiki/SciDB

fisherjeff• 9 years ago

They cover that in their paper: https://people.csail.mit.edu/stavrosp/papers/vldb2017/VLDB17...

jakebol9• 9 years ago

Jake from TileDB, Inc. In addition to the differences pointed out in the paper, I think SciDB and TileDB are very different philosophically. SciDB is architected very much like a traditional RDMS while TileDB is much more lightweight. SciDB encourages you to use their own query language (AQL), TileDB wants to integrate and extend the high level tools you already use (Python, R, etc.) with as little overhead as possible.

StanSeltser5• 9 years ago

Stanislav Seltser, Petacube kuwze: the following things are fundamentally different: license, software size,data model, sparsity support

SciDB --- license: Affero ,software size: 5GB ,data model: ACID ,focus on: dense ,dimensions: integer ------

TileDB license MIT ,software size <1MB , data model: eventual consistency via fragments , focus on: dense, sparse ,dimensions are integer, floats

sandGorgon• 9 years ago

Anyone know how it compares with Apache Arrow (that pandas is moving to) ?

http://wesmckinney.com/blog/apache-arrow-pandas-internals/

http://arrow.apache.org/

anilshanbhag• 9 years ago

Arrow is a columnar format. TileDB does clustering across multiple dimensions based. Arrow is great when you request specific columns. TileDb is great when you want rectangular sections of the space (think getting tiles from a map).

stavrospap9• 9 years ago

Stavros from TileDB, Inc. here: Arrow employs a columnar format to store objects like data frames. TileDB is also columnar (so you can sub-select attributes and perform analytics with very similar optimizations to Arrow), but the first-class citizens are (the more general) dense and sparse multi-dimensional arrays. Moreover, TileDB focuses on optimizing for the persistent storage backend, so that it can handle out-of-core analytics on massive datasets that cannot fit in main memory (e.g., genomics), or offer the same performance using less RAM (leading to cost savings in the cloud). Nevertheless, we are quite fond of Arrow, so we hope to work together at some point and integrate as seamlessly as possible.

polskibus2• 9 years ago

What do you mean by out-of-core?

acidflask• 9 years ago

How long until we can have Julia wrappers?

leethargo• 9 years ago

Since jakebol is involved in both TileDB and Julia, this might be up and coming?

Osmium2• 9 years ago

Looks exciting. What's the easiest way to get started with this via Python/NumPy? Looks like this is a design goal, but not currently supported.

stavrospap9• 9 years ago

Stavros from TileDB, Inc. here: We have elevated the Python/NumPy bindings as our top priority and have already started development. We will try to ship them asap. :)

Osmium2• 9 years ago

Excellent! Good to hear :)

Also saw in your documentation that you're concentrating on lossless compression right now, which makes complete sense. However, as a scientist, I just want to put a vote in for lossy compression too: it's not uncommon to work with large datasets given in float64 (because float64 is used for any intermediate processing steps), but that actual final precision we need to store is much less than that, but we're stuck with these huge binary files.

polskibus2• 9 years ago

How do I slice & dice TileDB with a query? Can TileDB compute aggregations or do I need to copy cells into main program memory out of TileDB and then perform all data manipulation?

stavrospap9• 9 years ago

Stavros from TileDB, Inc. here: Currently, TileDB is a storage manager, so it offers only efficient slicing. Support for aggregation queries (e.g., similar to what you can do with Pandas) is in our roadmap. We plan to work on it as soon as we ship the Python bindings.

sixdimensional2• 9 years ago

For posterity sake, it reminds me of MUMPS persistent sparse arrays, but built to scale much larger. This comment is not meant in any way to take away from this achievement, but rather to muse on where we were and where we're going.

It seems like the days we were stuck in particular or limited ways of thinking about databases/persistent storage are finally well and truly behind us! Now we have many awesome tools to choose from, better to have more tools in the belt than less.

jakebol9• 9 years ago

Haha, fully agree! It's good to separate MUMPS the language from MUMPS the storage layer / database which was innovative and pioneering in many ways. I don't think we need to re-hash MUMPS the language :)

sixdimensional2• 9 years ago

Agreed! :)

lowsenberg• 9 years ago

This looks very interesting. Currently we are storing our dense simulation (and experimental) data in NetCDF/HDF5. Given correct chunking, this seems to be pretty efficient both performance and compression wise. What would we gain using TileDB? How does performance compare with HDF5?

jakebol9• 9 years ago

Jake from TileDB, Inc. Performance wise I would look at the referenced paper in this thread which provides benchmarks for various workloads. As to what advantages TileDB may offer you that is problem dependent, esp. compared to dense simulation output data which is the use case HDF5 was designed for. If you have specific suggestions for ways to improve HDF5 for your use case we would love to hear about them.

stavrospap9• 9 years ago

Stavros from TileDB, Inc. here: HDF5 is a great software and TileDB was heavily inspired by it. HDF5 probably works great for your use case. TileDB matches the HDF5 performance in the dense case, but in addition it addresses some important limitations of HDF5, which may or may not be relevant to your use case. These include: sparse array support (not relevant to you), multiple readers multiple writers through thread- and process-safety (HDF5 does not have full thread-safety, whereas also it does not support parallel writes with compression - I am assuming you are using MPI and a single writer though, so still HDF5 should work well for you), efficient writes in a log-structured manner that enables multi-versioning and fault tolerance (HDF5 may suffer from file corruption upon error and file fragmentation - you are probably not updating, so still not very relevant to you). Having said that and echoing Jake's comment, we would love to hear from you how TileDB could be adapted to serve your case better.

A general comment: TileDB’s vision goes beyond that of the HDF5 (or any scientific) format. Considering though the quantities of HDF5 data out there (and the fact that we like the software), we are thinking about building some integration with HDF5 (and NetCDF). For instance, you may be able to create a TileDB array by “pointing” to an HDF5 dataset, without unnecessarily ingesting the HDF5 files but still enjoying the TileDB API and extra features.

TileDB: Storing massive dense and sparse multi-dimensional array data

Related Stories