Cassandra: Daughter of Dynamo and BigTable

http://www.insightdataengineering.com/blog/cass.html

by aouyang1 • 11 years ago

81 47 11 years ago

insightdataengineering.com

IndianAstronaut• 11 years ago

Article isn't loading.

dimi-31• 11 years ago

Pretty interesting read, the concept of tunable consistency was foreign to me, keep on the good work !

sciurus• 11 years ago

You may be interested in the Dynamo paper, which is what introduced it to me: http://www.allthingsdistributed.com/files/amazon-dynamo-sosp...

samplonius• 11 years ago

Riak has the same sort of concept: http://docs.basho.com/riak/latest/theory/concepts/Eventual-C...

I think Riak is richer in this area than Cassandra, because if the inconstancy can not be resolved, Riak can keep both versions and let you deal with it at the application level.

leef3• 11 years ago

What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook (DynamoDB from AWS is actually not based on Dynamo tech except in terms of what not to do).

[1] https://www.facebook.com/notes/facebook-engineering/the-unde...

edit: Added link to facebook post on hbase vs cassandra

themartorana• 11 years ago

What did Amazon move to? I wasn't aware they moved away from a Dynamo model.

bkeroack2• 11 years ago

Hulu uses Cassandra heavily, as does Netflix.

d_t_w• 11 years ago

Apple has 70k Cassandra nodes, if you send an imessage it's probably going into C* at some point.

threeseed2• 11 years ago

From what I heard Sony PSN uses Cassandra and EA uses it for many of their online services.

It really is the easiest database by far to scale that I've worked with.

vardump2• 11 years ago

Both Dynamo and Cassandra are written in Java. Are the replacements still Java?

At least Google's counterparts seem to be written in C++. Such as BigTable and GFS. Presumably also Spanner is C++.

In addition to C++, especially considering its less good safety/security record, Rust, Golang and Nim should be interesting alternative, safer, implementation language choices. In contrast to idiomatic Java, those languages provide significantly higher CPU cache hit rate for internal data structures due to no value boxing [1] and ability to reliably reduce problems like false sharing [2].

1: http://www4.di.uminho.pt/~jls/pdp2013pub.pdf (These issues can be worked around in Java by abandoning object orientation and instead having one object with multiple arrays (SoA, structure of arrays). In other words, not List or Array etc. of Point-objects, but class Points { int[] x; int[] y; ... } that contains all points.)

2: http://mechanical-sympathy.blogspot.com/2011/08/false-sharin... (False sharing performance issues)

threeseed2• 11 years ago

In what way is Rust, Golang, Nim safer than Java ? Rust I understand provides some nice semantics for thread safety but these exist in Java as world.

And I don't understand why anybody should care about CPU cache hit rate. The bottleneck is always going to be in the I/O pipeline. And Java is faster than C++ and vice versa in various situations.

jchrisa5• 11 years ago

Damien Katz (one of my cofounders at Couchbase and therefore has a dog in the fight) wrote this take-down of the Dynamo model, for folks wondering why?

http://damienkatz.net/2013/05/dynamo_sure_works_hard.html

TLDR quote:

The Dynamo system is a design that treats the probability of a network switch failure as having the same probability of machine failure, and pays the cost with every single read. This is madness. Expensive madness.

jedberg5• 11 years ago

I don't agree with his fundamental premise:

Network Partitions are Rare, Server Failures are Not

Network partitions happen all the time. Sure, the whole "a switch failed and that piece of the network isn't there anymore" doesn't happen a lot, but what does happen a lot is a slow or delayed connection, or a machine going offline for a few seconds.

noelwelsh2• 11 years ago

I think that's a simplistic analysis. What you want really depends on the workload characteristics. If you want an immutable data store with high write performance but infrequent reads the Dynamo model works quite well. Writes can be fast with a small quorum (you don't need to persist to disk immediately as you're relying on multiple machines not going down) and read consistency is not usually a huge issue for analytics tasks (who really cares if you miss an event in your processing?) and if it is you can usually afford to wait for quorum reads.

bkeroack2• 11 years ago

Reads absolutely must go through the consensus system (whatever it may be) if you make any guarantees against stale reads. Yes it's expensive, but that's why there's no such thing as a magical distributed database that scales linearly while providing perfect consistency guarantees all the time.

strmpnk3• 11 years ago

The redundant work has a drastic improvement on tail latency. This is both true in a health cluster but even more true once even small failures or fluctuations occur. You might be able to tune your network and your software to _minimize_ observed partitions but you are very unlikely to see a system free of fluctuations outside of the hard-realtime system construction which none of the above qualify for by a very long shot.

It's a heavy price, sure, but in return you'll be able to round off the tail in most scenarios. This is a trade off that many don't properly consider when they try to focus on that mean performance while they miss part of the point of the dynamo model which helps provide better guarantees about how things perform in more cases, including highly tuned clusters with very few major failures.

noelwelsh2• 11 years ago

It's not so simple. The main complaint I have heard is that programmers find it difficult to deal with eventual consistency exposed through vector clocks. It's only recently that CRDTs have been reasonably well known, and they solve this problem. Riak 2.0 includes a CRDT library but it might be too late.

Note that Cassandra doesn't actually handle eventual consistency properly, and has weird corner cases a result (e.g. it's infamous "doomstones"). As an immutable data store it works very well, particularly when you have a high write load.

lobster_johnson2• 11 years ago

I found Riak's CRDT implementation disappointing. They require that you define them beforehand in a schema, which defeats much of the point of using a schemaless database in the first place.

flurdy• 11 years ago

That is a good thing. It does not mean that Dynamo and Cassandra are bad choices, but that companies and teams can move onto other solutions when their requirements changes. In an Open Source model just because you invented it does not mean you are stuck with it.

Not sure I could work in a one product company where you have to eat your own dog food in all situations even where it really is not suited for.

akbar501• 11 years ago

What's interesting to note is that Dynamo/Cassandra usage was killed at both Amazon and Facebook

IMO, the only way use, or former use, is interesting is in understanding why a specific company moved to, or away from, a given technology.

Specifically, what was the original use case that was the basis for original use?

Why did the company choose to change technology? Did the use case change? Did the technology fail to satisfy the original requirements? Did a new technology with substantially better capabilities emerge? And so on?

Simply stating that company X uses Y (or company X no longer uses Y) does not provide a lot of information that other companies can use as the basis for their decision.

digitalzombie• 11 years ago

Cassandra is a pretty good NoSQL database. I'm a big fan of it.

Of course if your data change often and very quickly you shouldn't be using Cassandra. You'll end up tombstone death. Deletes are not real delete, they're soft and just have a timestamp that eventually will be deleted (tombstone). Every delete creates a tombstone, if you're hashkey/column key have 50 tombstones, it have to go through those tombstones before getting the values. The reason is some trade off for faster write but shitty read if you update a lot of the same key.

Overall, one way of thinking of it is a souped up hashkey db that can do some relational queries which is much more than the usual hash key noSQL. Seeing how it's a hash key type database you can see the trade off of Cassandra and it's siblings versus say MongoDB or CouchDB.

I used it for time related stuff that has simple data. It's mostly immuatable so cassandra was perfect for it. Example is a daily tv show release date.

jermo2• 11 years ago

Increasing the size of memtables should reduce the tombestone overhead since entries will get overwritten in memory.

salex89• 11 years ago

Sometimes I feel that the Cassandra row length limit is not taken seriously enough. If you have some high frequency data it will not take too long until you hit the limit. Also, the guide states that you should not really attempt to get it that far. I'm not sure if schemes like the one in the example are future-proof. In a couple of years developers will start hitting the limit and we'll see mailing lists with postmortems :) . A bit apocalyptic, but I believe we should find an idiomatic solution for it. I use rounded time periods as a part of the partition key. Yes, it makes the data a bit more complicated to query, but I have not thought about a better solutions.

Time series databases like KairosDB are quite good, but for simpler data structures and something describable as a metric. Also you may face issues introducing relatively less known software, and get locked in, in your company.

Cassandra: Daughter of Dynamo and BigTable

Related Stories