Call me maybe: Cassandra

http://aphyr.com/posts/294-call-me-maybe-cassandra

by lucian1900 • 13 years ago

268 51 13 years ago

jbellis7• 13 years ago

The bugs he ran into with lightweight transactions were fixed within days, two weeks ago, and included in the 2.0.1 release shortly afterwards.

itp2• 13 years ago

The fact that these bugs were so trivially reproduced and made it into an initial release at all should be cause for concern.

Conflict of interest disclosure -- I work for FoundationDB, where we put a shockingly high level of effort into testing our software in simulation and the real world. [1][2]

[1] https://foundationdb.com/white-papers/testing

[2] https://foundationdb.com/blog/quicksand-continuous-real-worl...

j_baker• 13 years ago

Please. I have yet to work in any environment that hasn't had a brown bag release or two, regardless of how well tested it is.

Nice marketing attempt though.

argvzero2• 13 years ago

advice from someone who's been at this nosql thing for about 6 years now. schadenfreude will always come back to bite you. tread lightly.

cbsmith4• 13 years ago

The fact that these bugs were so trivially reproduced and made it into an initial release at all should be cause for concern.

Umm... no. The .0 releases of the project are actually where you'd expect the most bugs. That's where you have the first commit with an implementation of a new design.

You'll note that DataStax's commercial version of DSE is still based on 1.2.x...

rdtsc• 13 years ago

If foundation db released and is it open sourced?

I remember others (employees of company) hailing its wonderful qualities for quite a while now (years), then I go to the website and all I could find was a bunch of white papers and a registration form. And here it seems a bit of a "my vaporware's features are better than your shipped product's features".

Now matter how many white papers there are I would still put my data in Cassandra rather than this new thing (last I checked I couldn't even download it, I had to fill out a form of some sort).

tel2• 13 years ago

DataStax still claimed these properties while the bugs were live. It's important they fixed them quickly, but the point still stands—the point of the entire series, I think—that database companies very frequently make big claims that few people go through the process of checking... even at the companies themselves.

jbellis7• 13 years ago

It's important to distinguish architecture and design from implementation. The former may be correct, even if the latter is found to have a bug.

cbsmith4• 13 years ago

Have you ever seen a product feature description edited on account of a bug report, particularly one that was resolved within days?

threeseed• 13 years ago

You mean companies sometimes promote features that still have bugs ? My god that is incredible. I can't believe that has never happened before. Oh wait it has. On nearly every single software project in the history of software projects.

dilap• 13 years ago

Of course, if aphyr hadn't reported the bugs before publishing, they'd be live right now, right?

jamesaguilar2• 13 years ago

And who knows how many more are lurking? I think that's the issue exposed by this post -- if these really basic things are broken, why would anyone believe that the actual tricky stuff has been tested?

jbellis7• 13 years ago

It's awfully hard to prove a negative, but our test suites are open [1] [2], and so is our bug tracker [3].

[1] https://github.com/apache/cassandra/tree/trunk/test/unit/org... [2] https://github.com/riptano/cassandra-dtest [3] https://issues.apache.org/jira/browse/CASSANDRA

cbsmith4• 13 years ago

I think that's the issue exposed by this post -- if these really basic things are broken, why would anyone believe that the actual tricky stuff has been tested?

So, I think it is relevant that LWT is still a very new feature in Cassandra and not something basic to it at all (arguably counter to a lot of its original design goals).

Personally, I was much more concerned by the server side timestamps only using millisecond granularity (and even that is somewhat understandable given the JVM's limitations).

tomjohnson3• 13 years ago

i've been following kyle's jepsen project for a few months (check out his other posts) - and it seems (thankfully) to be nudging the focus of discussion away from only "performance metrics" (transactions/sec, etc.) of distributed systems and toward data consistency and behavior in the face of partitions and other faults. ...what's the use of being able to make 20k updates per second if half your data is lost during a common failure?

kudos!

fleitz3• 13 years ago

Great article on the fundamental problems associated with mutable state. The fundamental problem is that the idea of an object with a set of state that is the same to all observers violates pretty much the whole of information theory. It's not a problem that will ever be fixed with out changing the fundamental laws of the universe.

Ditch the mutable data and you can stop asking questions like what do we do if 10 becomes 10.5 before it becomes 11 and start storing values which never change.

aphyr3• 13 years ago

The question of consensus and mutability are, to some extent, separable; immutable data does not provide liveness without consensus guarantees. See, for instance, Datomic's design, where a strong coordinator is needed to serialize updates to otherwise immutable/replayable state.

mtrimpe• 13 years ago

Indeed. Although the use of confluent data structures where possible combined with immutable data stores combined and 'live' functional reactive programming across device boundaries -would- drastically simplify cases like these.

fusiongyro4• 13 years ago

The argument here is basically to replace mutable state with event sourcing. It's an interesting idea and sometimes the right one, but if each user action that triggered a one cell update becomes an event I have to keep forever, I see my database size exploding. This is also going to yield performance problems I'll be tempted to avert with caching, leading to fantastically bad performance or cascading failures whenever the caches fail or are restarted.

I'm sure it's the right answer for some people, some of the time. In fact, I'm sure it's not applied as frequently as it should be. But it's definitely a specialized tool with special applications, where mutable state is, for better or worse, the hammer we can and ought to continue relying on.

agilord2• 13 years ago

but if each user action that triggered a one cell update becomes an event I have to keep forever, I see my database size exploding

I think there might be a balance here: you can always garbage-collect events that are already merged in an updated value of the given object. Depending on the requirement, this GC can be done e.g. after days or after months of that merge...

dragonwriter• 13 years ago

The argument here is basically to replace mutable state with event sourcing. It's an interesting idea and sometimes the right one, but if each user action that triggered a one cell update becomes an event I have to keep forever, I see my database size exploding.

Once an event is visible to all users, it can be merged into the base state of the system and no longer needs to be stored (or at least kept online) as a separate event. For the performance reasons you allude to, you probably want to do that as much as possible.

(This is still effectively an append-only series of immutable states, but losing, at least from regular on-line access, "older" states that no one can see anymore.)

fleitz3• 13 years ago

if each user action that triggered a one cell update becomes an event I have to keep forever, I see my database size exploding

Yes I could see that getting prohibitively expensive when SSDs cost 70 cents/GB and hard drives 5 cents/GB. You should really throw out your historical data at those kinds of costs, probably not worth 5 cents per GB.

Personally, that's why I don't keep backups, files change all the time, and I was going broke making sure I had older copies of my data. I'd rather just rewrite all my code and retake all my pictures.

oakwhiz• 13 years ago

As the other commenters have pointed out, I'm pretty sure that you're supposed to compact old events once they have been determined to be eventually consistent amongst all (or almost all) of the nodes.

jerf• 13 years ago

Information theory does not have a problem, only, as you say, our universe. Mathematicians and their various hangers-on like programming language researchers often prefer to deal with models that have no concept of time, in which the very concept of "observer" is extraneous since there isn't really anything like a "point of view". Everything just... is.

Also, I think the focus on immutability misses the more interesting discussion about lattices and how important they are to distributed programming. Check out this video, which will take you up from the beginning: http://vimeo.com/53904989

(Seriously, if you do anything remotely distributed, this is required viewing. There's some serious stuff going on in the distributed world and this is a great intro.)

marshray• 13 years ago

The talk is getting interesting at 11 minutes in. But I can't find the slides anywhere and the video was edited by someone who likes to show the slide for 5 seconds and the person pointing at it for 2 minutes.

penguindev2• 13 years ago

I'm concerned that the paxos test couldn't handle 50/sec without timing out practically all of them. Also, the 'official' datastax blog post I read describing the feature was not clear at all describing the granularity / sharding of the paxos state machines. It said a 'partition'. Does that mean a 'vnode' in cassandra speak?

Otherwise, kudos for adding a potentially useful feature to cassandra.

jbellis7• 13 years ago

As I mentioned above, we fixed the bugs he ran into; this is not representative behavior of 2.0.1.

The granularity is a CQL partition: http://www.datastax.com/documentation/cql/3.0/webhelp/index....

penguindev2• 13 years ago

Interesting findings re. the paxos bugs. Dare I ask how reliable cass's sharded paxos is (in theory, at least) while you're adding/removing nodes from the cluster? How do all clients see consistent membership?

jbellis7• 13 years ago

The principle is the same as with normal reads; expand the quorum to include both old and new owners until the transition is done: https://issues.apache.org/jira/browse/CASSANDRA-833?focusedC...

dschiptsov• 13 years ago

What is the reason for trying to squeeze into JVM an engine which must be implemented on the one level up - an OS level? ,)

Is it still not obvious that "universal object storage" is a naive idea?)

Call me maybe: Cassandra

Related Stories