Thinking in Datomic

http://pelle.github.com/Datomic/2012/07/08/thinking-in-datomic/

by pelle • 14 years ago

96 29 14 years ago

pelle.github.com

mbreese2• 14 years ago

Couldn't you just model your data in a traditional RDBMS with a time stamp as well? In fact for mutable data that you'd like to keep old versions of, this is pretty standard. A simple design would be to have a separate table for person_locations that mapped a person to a location.

With everything else, the standard RDBMS table could be considered as having a 'snapshot' of the Datomic values.

I'm still not sure what benefit this has over a traditional DB. Perhaps I'll just have to wait for the next post.

jcoene• 14 years ago

Of course you can make something similar if you're a) willing to build it and 2) don't care about the tremendous amount of complexity you're adding to your application in order to do so.

Both of those sound like bad deals if there's a packaged solution like Datomic that's built specifically for the use case.

pelle4• 14 years ago

You can do it all yourself if you want, but Datomic does it for you without you having to do anything.

Also as it does it on a datom level it is a lot more efficient than versioned rows.

Interestingly one of the early selling points of Postgresql was this time travel functionality. But it was yanked out in 6.2

http://www.postgresql.org/docs/6.3/interactive/c0503.htm

mbreese2• 14 years ago

Yes, but you also loose the flexibility of not doing it. I'd rather work in an established RDMBS and configure in the design I'd like to use rather than use a database that requires a specific configuration.

I'm sure that Datomic has it's place, but the examples you gave in this post aren't that convincing.

mattdeboard• 14 years ago

The difference between Datomic and RDBMS isn't JUST a timestamp FYI. In fact I'd expect/hope this blog post is a just a prelude to Datomic's transactional model which is the real differentiator.

pelle4• 14 years ago

Absolutely. That is where I'm going in the next article. The transactions are extremely cool.

snorkel• 14 years ago

Yes, you can do similar in a traditional RDMS, you have a table called Person and another table called Person_properties with a schema of pid, property, value, timestamp. The object model for Person would have a save() and load() method that knows which properties belong to the base table (base properties) and which properties spill over in into the triple table. The separate properties table has the convenience of stashing arbitrary persistent properties in a Person without needing to migrate the underlying schema or change the base definition.

mtrn2• 14 years ago

The temporal aspect seems to be somewhat ignored by mainstream db development - in some fields, e.g. BI you'll have slowly changing dimensions; and the event sourcing pattern promises a time-machine view on data.

Since I'm writing a history-aware application at the moment, I recently looked into different patterns for this and trying a mixed strategy at the moment (SQL DB used as document store and a single event log, that accumulates changes - a lean approach though, a few hundred lines python for the data access layer; what always gets ugly is the validation, which your application must take care of).

I wish, there was more hands-on material on the subject (some resources dive depth into bi-temporal modeling, but I feel your schemas can get complex (= expensive) very fast).

dustingetz• 14 years ago

by my understanding, datomic's persistence[1] layer is just a big huge persistent[2] data structure. there is tons of literature, books, papers, etc about persistent data structures in the functional programming community.

[1][2] 'persistent' means different things here.

calebmpeterson• 14 years ago

So, Datomic is a persistent data structure persisted to nonvolatile storage... I've wanted one of these for at least the last 2 years.

If only there was an open source implementation of this concept that I could run on my own hardware. Does anyone know of such a beast?

And, no, I don't want to roll my own versioned/timestamped row schema in an RDBMS - been there, suffered that.

mtrimpe4• 14 years ago

Big Data by Nathan Marz is a Manning Early Acces book (i.e. not fully written yet) that explains how to hand-roll your own custom Datomic style DB in Hadoop/HBase.

It might be of interest to you given your current project.

mtrn2• 14 years ago

Thanks for the tip, will take a look at it. Probably after this project, the current app must be lean (no big dependencies, just install a database scheme, pip install and go).

beagle3• 14 years ago

kdb+ is head and shoulders about the rest when it comes to introducing time / order into relational data. It's lean, minimalistic and ridiculously fast.

Unfortunately, it is also ridiculously expensive.

sbmassey• 14 years ago

It's also easy to bring to its knees with seemingly innocent queries, and has a pretty darn opaque programmable interface in Q. (OK, the vaguely SQL-like stuff is not too hard, once you understand the differences from normal SQL, but doing anything lower level than that is awesome/crazy)

EzGraphs• 14 years ago

For what it is worth: http://en.wikipedia.org/wiki/Temporal_database

A more extensive description/case study of bi-temporal database design: http://www.cs.arizona.edu/people/rts/tdbbook.pdf

This case study tried to be somewhat rigorous, but from what I recall the SQL did get complex and expensive pretty quickly. There are a few attempts to create products to mask the complexity as well:

http://www.timeconsult.com/Software/Software.html

For living in time and dealing with it regularly in a common sense sort of way, it is really rather challenging to manage effectively as data.

jerf2• 14 years ago

There have been numerous triple stores. Mozilla/Firefox even used one as its core backend for a while before ripping it out and replacing it with SQLite. Why do you think those failed, and how is Datomic going to avoid the same failures?

That is, I've seen this pitched as the solution to all our data woes numerous times now, why is this time different?

mtrimpe4• 14 years ago

It's not a triple store. It's closer to an OODBMS focused on persisting large object graphs with an immutable append-only datastructure, thus maintaining full history and being quite scalable with a single-writer many-reader configuration.

pelle4• 14 years ago

It is actually implemented more like a triple store than an oodbms. Just with time added as a 4th parameter.

Their Entity class makes it feel a bit like an oodbms though in that it kind of acts like a hash map.

irickt• 14 years ago

Triples are a simple data model that can be used for web-like graphs, relational tables, and hierarchical hashes. This is needed irrespective of Datomic's prospects. Just look at the the current massive adoption of linked data. http://news.ycombinator.com/item?id=3983179

damncabbage• 14 years ago

"Datomic is so different than regular databases that your average developer will probably chose to ignore it."

Playing to the Well I'm obviously above average gut feeling. Cute.

Thinking in Datomic

Related Stories