jrullmann

Take a look at the Overview slide of the presentation, where this information is listed for each database the author discussed: http://image.slidesharecdn.com/newsql-2015-150213024325-conv...

Seeing how each visualization adjusts as I change the original dataset is so useful. The technique reminds me of Bret Victor's amazing work.

Ladder of Abstraction Essay: http://worrydream.com/#!2/LadderOfAbstraction

Stop Drawing Dead Fish Video: https://vimeo.com/64895205

This is awesome, thanks for sharing!

Along those lines, this article about Malcolm McLean (often called the father of containerization) is a great read.

http://hbswk.hbs.edu/item/5026.html

Looks cool. Would love to know what Conspire uses as their technology stack.

Nice utility. I submitted a pull request for FoundationDB SQL Layer https://github.com/emirozer/fake2db/pull/4

Just submitted a pull request for FoundationDB: https://github.com/ept/hermitage/pull/1

Very cool - would love to see tests of FoundationDB!

FYI I'm an engineer at FDB, happy to help.

Great article. A lot of engineers don't have personal experience with these kinds of network failures, so sharing stories of their consequences means more engineers can make informed (and conscious) decisions of how much risk can be tolerated for their applications.

One thing that you could gleam for this article-and I think that this is incorrect-is that the application or operations engineer is responsible for understanding the nuances of distributed systems. In my experience the number of people who are relying on distributed systems is much larger than the number of people who understand these issues.

So what we really need are systems we can build on whose developers understand how to build (and test!) the nuances of data convergence, consensus algorithms, split-blain avoidance, etc. We need systems to gracefully-and automatically-deal with and recover from network failures.

Full disclosure: I'm an engineer at FoundationDB

What kind of consistency do you expect to provide with this future syncing feature? I assume it will be eventually consistent. Is that right? How will conflicts be resolved?

That's a pretty unusual setup :)

I'd be interested in a blog post about how you choose this architecture.

Out of curiosity, what database are you using to store the data?

The author says that different data stores are good at different things, so we need to use multiple data stores in our application. He proposes that a data service layer can abstract these implementations, making it easier to swap out data stores as needed. I think that's a good idea. Separating our applications from data store specifics is a big reason why we use ORMs and ODMs today.

However, I think there are challenges with this polygot data store architecture that he doesn't address. Each addition requires due diligence to understand its CAP trade-offs (which the author mentions briefly), scalability and performance characteristics, how to configure, etc. These are non-trivial concerns even for a single database. It's important to consider these challenges when building out a data store or data service.

I'd propose an architecture where the data services layer itself exposes different data models to the application, all of which are persisted in a single data store. Given that many data stores use a key-value store under the covers anyway, translating the specific data model down to a single, persistent data store would simplify operations while exposing the desired data model to the application. (As a caveat, this multi-model approach requires ACID transactions to ensure strict consistency when translating between data models.) This approach provides operational simplicity with just one data store and application efficiency by exposing the "right" data model API.

Full disclosure: I'm an engineer at FoundationDB, a database that provides ACID-compliant polygot data models on a single key-value data store (http://www.foundationdb.com).

Makes sense to me. Thanks for the feedback.

That's fair. My runner up title was Scaling All the Things: An Honest Look at Limited and General Transactions. Better or worse?

Hey guys, I'm the author of the blog post. I'm happy to answer questions about performance results, argue about the necessity of acid transactions, or discuss anything else in the post.

I've been playing around with the idea of doing a couple of these posts - for example, I could do one with multiple datacenters. This would be a good scenario too.

Thanks for pointing this out. That's totally a mistake on my part. I've updated the table to include the CRDT result, showing Riak's 0% data loss.

Good idea - I've updated the blog post. Thanks for the feedback!

Right. Most of the time you would simply retry the transaction - that's what FoundationDB's automatic transaction retry loop does. The first two tests I ran do that, but the third (foundationdb-append-noretry) does not.

Thanks - it was a great way to learn more about the database, and the effect of network partitions in general.

You'll need two pieces to run Jepsen against a database:

1. Code to provision a five cluster database (using Aphyr's Salticid deployment system)

2. Code to do some reads and writes to the database (this is run by Jepsen)

It might be helpful to look through the pull request I submitted to Aphyr for FoundationDB's test, so you can see the pieces: https://github.com/aphyr/jepsen/pull/10

Hey guys, I'm the author of the post. I'm happy to answer any questions or comments you may have about it.