FoundationDB Fault Tolerance Demo Video

http://blog.foundationdb.com/post/50924738464/foundationdb-fault-tolerance-demo-video

by nlavezzo • 13 years ago

32 15 13 years ago

blog.foundationdb.com

nlavezzo6• 13 years ago

At 4:40 in the video we do a network partition - which seems to be relevant today in light of the aphyr / Jepsen post.

Also covered are single and multiple machine hard power failures, rejoining the cluster, and total cluster power loss and recovery, all in one take.

Happy to answer questions.

ericingram2• 13 years ago

Question: Does FDB have any sort of "find" method in its API? As a key/value store I understand that I can set/get keys, but is there any way to find keys based on nested values?

I.e. fbd.find({someproperty: "123"})...

misframer• 13 years ago

No, but you can utilize some data modeling techniques, such as the ones they mentioned on their website [0].

[0] http://foundationdb.com/documentation/beta1/data-modeling.ht...

voidmain• 13 years ago

A database is going to do a query like that in one of two ways: scan over (all of|a column family of) the data, or look in some kind of index on "someproperty".

In our approach this kind of functionality is the responsibility of a higher "layer" of the system above our key/value API. There are lots of ways of organizing and indexing data with different tradeoffs between search and update performance for different kinds of queries. We support pretty much all of them-- not by building any of them into the core, but by providing a general transaction facility that permits arbitrary indices to be updated.

For example: in a database like Mongo, queries like that one don't scale linearly even if there is an index, because you have to ask every shard to look in its local index. In FoundationDB, you would store the index in the database as keys like (someproperty_index, "123", _id) and an index read would only need to go to a single server.

It can be a hassle to create all the higher level functionality yourself, though! Over time, we and others will be offering more "opinionated" layers to save you time.

ericingram2• 13 years ago

What a great demo! We're currently building a critical service on MongoDB and feel a bit leery about some of the issues others have expressed at scale. Would be interested in testing FoundationDB for our purpose in the near future.

nlavezzo6• 13 years ago

Thanks! We thought since it was a hit in person we should share it a bit broader.

You mention scale - do know that what you're seeing in the video is probably the weakest FoundationDB cluster ever assembled :) Our in-house testing cluster is a 48 machine, 192 core setup, and it works great and scales almost linearly. You can see our performance numbers at www.foundationdb.com/performance (though those results are outdated now, the tests were on a cluster half the size and several versions ago).

If you'd like to chat about your particular use case, send me an email at nick at our domain.

burgreblast2• 13 years ago

Great demo. Really curious about the nuts and bolts of the set up. What OS, and what is actually launching at startup? Again, very nice job.

nlavezzo6• 13 years ago

I'd have to ask one of the guys that put it together for specifics in the morning, but the OS is Ubuntu Linux, and the applications auto launching are of course the FoundationDB server which then connects to the rest of the cluster and begins doing work, and also a client with a Firefox browser running the testing / demo app which increments the counter, changes the circles, and uses (I think) HTML 5 to render the images.

SirSkidmore• 13 years ago

Kind of irrelevant, but just out of curiosity, but what are those little PCs and monitors? They look very interesting!

burgreblast2• 13 years ago

Ah, I see. Demos are hard but you did a great job. Very well done.

dangerboysteve• 13 years ago

Really interesting talk and I also just watched your "NYC Database Month" video.

What are the node costs in your licensing model ? Also any plans to release Flow as open source ?

nlavezzo6• 13 years ago

Thanks. We haven't published pricing yet but we are aiming to make it in the ballpark of the hardware & operational costs of the machines required to run a particular size cluster. I.e. if you are running a FoundationDB cluster at 50% load then the licensing costs shouldn't be out of sync with the costs of owning and operating, or renting, the machines in the cluster.

It does look likely that we will open source Flow at some point, but it will take a good amount of work to do it properly and right now there is plenty of it to go round. Always good to hear interest for it though!

djtriptych• 13 years ago

Amazing demo. I'd love for aphyr to take a closer look...

FoundationDB Fault Tolerance Demo Video

Related Stories