A Tour of Amazon's DynamoDB

http://www.paperplanes.de/2012/1/30/a-tour-of-amazons-dynamodb.html

by timf • 14 years ago

143 30 14 years ago

SaltwaterC3• 14 years ago

The API doesn't bother me as much as the temporary authentication via STS. Temporary credentials for database access? Seriously? I am the only one who sees how ridiculous this is?

amock2• 14 years ago

What's wrong with using temporary credentials?

SaltwaterC3• 14 years ago

They add useless latency and another point of failure. And sometimes, the AWS APIs DO fail.

jedschmidt• 14 years ago

I've been waiting for Mathias to write something like this, and I'm glad he didn't waste any time diving in. For those who don't know him, he has more breadth in the area of emerging datastores than anyone I know; he literally wrote the book on Riak[1].

I've also been acquainting myself with the DynamoDB API over the past week, and am building a node.js binding that I hope will abstract away most of the esoteric aspects of interacting with it. It has full API coverage currently and is tested in Travis[3], so now I'm writing the high-level interface. So far I've covered about half of the operations DynamoDB offers, but would love to hear any ideas/feedback.

    [1] http://riakhandbook.com/
    [2] https://github.com/jed/dynamo
    [3] http://travis-ci.org/jed/dynamo

ryanfitz• 14 years ago

Looks like we are building a very similar node.js client for dynamo. https://github.com/Wantworthy/dynode

jbellis2• 14 years ago

I'm still struck by what a strong endorsement DynamoDB is of the design decisions we've been making in Cassandra over the last couple years. Composite keys, distributed counters, ...

More details: http://www.datastax.com/dev/blog/amazon-dynamodb

j2labs• 14 years ago

Didn't you folks design Cassandra after the Dynamo paper though?

thamer• 14 years ago

Composite keys (multi-dimensional) and distributed counters were not in the Dynamo paper. They were added to Cassandra, which is indeed based partially on Dynamo.

stonemetal• 14 years ago

Yes, but the Dynamo paper offers two operations Get(key) and Set(key, value). Anything Cassandra adds atop that they did themselves i.e. column families, secondary indices.

amix• 14 years ago

Cassandra initial team was lead by Avinash Lakshman who was one of the authors of Amazon's Dynamo. So it's not strange that Cassandra is hugely inspired by Dynamo (and vice versa).

chargrilled• 14 years ago

How does Cassandra stack up against HBase these days?

Both projects seem to be moving so quickly that it's really hard to find an up to date comparison.

leef• 14 years ago

Dynamo as described in the original paper was not made into a web service.

Facebook took up the Dynamo torch and created Cassandra... then switched to HBase after trying to use Cassandra in production.

Have you ever stopped to wonder about what Amazon might have taken OUT of Dynamo before launching DynamoDB? Things that Cassandra still has and just might be holding it back?

That to me seems much more important than just saying "DynamoDB has feature X that Cassandra has had for years! We're so smart!"

jbellis2• 14 years ago

There's less to write about there... One of the first Cassandra decisions was dropping vector clocks, reasoning that vector clocks weren't worth the additional complexity for 99% of uses (once you move from key/value to rows + columns). DynamoDB also switched to items/fields and dropped vector clocks (or at least does not expose them).

The original Dynamo was plain key/value with O(1) routing and vector clocks; there's not much else to strip out. :)

P.S. Facebook built Messages on HBase instead of Cassandra for political reasons rather than technical, and shards HBase to mitigate the availability problems it has otherwise. Facebook never ran an Apache Cassandra release in production.

foobarbazetc• 14 years ago

I'm struck by how you somehow manage to pimp both Cassandra and DataStax on every post which has something to do with databases and/or datastores. :)

taylorbuley• 14 years ago

Does anyone know in which language DynamoDB is implemented? I've read somewhere that SimpleDB is done in Erlang. Is that the case with DynamoDB as well? I've been reading about Ets and Dets in Erlang and it makes me wonder whether they have anything to do with either of these data stores.

gigq• 14 years ago

I haven't seen it written anywhere, but judging by the __type in errors like this, I would say Java:

  {"__type":"com.amazon.coral.validate#ValidationException",
  "message":"One or more parameter values were invalid:
  The provided key size does not match with that of the schema"}

cleaver2• 14 years ago

DynamoDB is not open source, at least according to a blog comment by Jeff Barr of AWS. (http://aws.typepad.com/aws/2012/01/amazon-dynamodb-internet-...)

I'm certainly not going to commit to platform lock-in like that. I know a lot of folks were hit hard by Googles app engine price changes and it could happen again here with DynamoDB.

techscruggs• 14 years ago

Then you don't use S3?

sciurus• 14 years ago

There's an open implementation of S3.

http://open.eucalyptus.com/wiki/EucalyptusStorage_v1.4

cleaver2• 14 years ago

I don't actually, but I feel there is less lock-in. It wouldn't be hard to move off.

Also, the risk of S3 pricing changing greatly is constrained by the wide adoption. If DynamoDB failed to become pervasive, it would be relatively easy for Amazon to increase the price to make an underutilized service profitable. OTOH, if DynamoDB does become as popular as S3 your exposure to risk will be much less.

antirez2• 14 years ago

About pricing, it is interesting how caching is apparently not an effective option to reduce the bill in a significant way since data size is a big component of price.

justincormack• 14 years ago

Thats because it is SSD based, so size starts to matter again.

antirez2• 14 years ago

Yes but real hardware with an SSD disk plugged inside could deliver excellent performances with many reasonable on-disk databases, with a fraction of the cost per megabyte, with many many millions of keys.

It is like that other than the fact you don't have to manage the DB, that is a good point, here the best reason for many small-mid business to use such a service is since they are already on EC2. May work in the short term but I see something odd about this model.

siculars2• 14 years ago

I would second Mathias' thoughts in that this does feel more bigtable/column oriented than traditional dynamo. Dynamo and its derivative, Riak (not from amazon), make no attempt to determine data types or schema in any way. The fact that dynamodb can do range scans and has various counting features leads me to believe that it is more hybridized along the lines of Cassandra. Either way, it is a welcome addition to the nosql toolset that developers have to chose from today. I will certainly look at it more closely.

rbranson2• 14 years ago

Actually Riak can be used in a way in which it is content-aware of JSON and XML documents.

http://wiki.basho.com/Riak-Search---Indexing-and-Querying-Ri...

siculars2• 14 years ago

Search and Indexing were addons to the initial Riak product that were born from user demand. Only recently (1.0 I believe) was Search integrated into the main Riak distribution. Riak has no capability of updating values in place, ie. a partial update and no means of ordering or pagination. Some of these features are available via mapreduce queries or Search/Index as you noted.

I'm by no means harping on Riak. I actually use it on a number of projects. But reading about DynamoDB's capabilities does not conjure Riak, it conjures Cassandra and HBase.

A Tour of Amazon's DynamoDB

Related Stories