FoundationDB: A new generation of NoSQL database

by shansinha79 • 14 years ago

78 73 14 years ago

Without wanting to be any snarkier than usual, all-the-scale-without-the-compromise has been possible for decades. It's just really expensive.

shansinha79• 14 years ago

good point.. the key difference here is that it's also "..without the cost"

olalonde• 14 years ago

How?

d0ugal3• 14 years ago

noSQL and YesACID. It's that easy.

jacques_chester2• 14 years ago

Call your local Oracle, IBM, Sybase, Teradata, AsterData etc sales engineer. They'll be happy to fill you in.

Be prepared for sticker shock.

aaronblohowiak2• 14 years ago

Explain how you can have availability in the face of network partitions and a perfect total ordering to events (ie: the sequentiality you claim.)

jamesaguilar• 14 years ago

Presumably they sacrifice availability? If your network is only partitioned 1/1000 of the time and you only need 99.9% uptime, that would be a viable approach.

DanWaterworth2• 14 years ago

You can use Paxos. There will be some partitions that can cause an outage, but this is only the case when there is no majority of nodes that can communicate with each other, which in practice means you are exposing yourself to a vanishingly small risk.

aaronblohowiak2• 14 years ago

3 rounds of communication to reach consensus for every transaction? Off the cuff, it seems like that would hinder the performance. I haven't tested it, though..

sausagefeet2• 14 years ago

http://cs-www.cs.yale.edu/homes/dna/papers/calvin-sigmod12.p...

surrealize• 14 years ago

That system works by giving up interactive transactions. But the foundationdb page claims to have interactive transactions, so it must be doing something else.

From Abadi's blog:

http://dbmsmusings.blogspot.com/2012/05/if-all-these-new-dbm...

Calvin requires all transactions to be executed fully server-side and sacrifices the freedom to non-deterministically abort or reorder transactions on-the-fly during execution. In return, Calvin gets scalability, ACID-compliance, and extremely low-overhead multi-shard transactions over a shared-nothing architecture.

The comments on that post are pretty interesting, too.

And from the foundationdb features page:

http://foundationdb.com/#features

FoundationDB transactions are true interactive sessions, unlike distributed databases that require stored procedures. This means that client code can make an iterative series of reads and writes over the network to execute complex transactions.

blackhole• 14 years ago

This very badly needs proof that it is, in fact, ACID compliant. Incredible claims require incredible evidence.

Dave_Rosenthal2• 14 years ago

FoundationDB co-founder here.

Yes, we provide the strongest level of ACID semantics. Although proving things about large computer programs is pretty hard, we have spent much of the past three years building testing and validations systems to ensure this is true. We run tens-of-thousands of nightly simulations to check system correctness and properties in the face of machine failures, partitions, etc. We also run long-running tests on real-world clusters using programmable network and power switches to simulate these same cases in the real world.

So, we've convinced ourselves. What would you like to see on the site to help provide the kind of incredible evidence you're looking for?

tagx• 14 years ago

I worked under two of these guys two summers ago. They're really sharp guys and I have faith they can back up their claims.

shin_lao4• 14 years ago

I'd be surprised to see how they can be truly ACID and truly scale out.

This is an unsolved problem to me, I submit there are fine prints regarding ACID or scalability...

wmf4• 14 years ago

Jim Gray solved it in the 80s, but then we forgot about it.

shin_lao4• 14 years ago

To which of Jim's contribution are you referring to? I wasn't aware he designed a scale-out ACID transaction principle.

fdr6• 14 years ago

Hmm. Okay. I open a transaction for three days (let's say indefinitely), maybe reading a few tuples here and there and updating a few, but never committing. How does this system not fall over? I guess it could abort my transaction...

Still, I think their performance numbers can be legit, because they are much, much slower than what a handful (much less than 24) machines could do if fully partitioned and without coordination. 500KRead/Second on such small values is not really fantastic performance over 24 machines. I also don't understand the initial burst-capacity on the read-side either, although I can guess -- on writes it can make sense because some work is deferred, but I'm trying to understand how that can happen on reads as well. My guess is a transaction ID allocation on-read that hits a wall somewhere.

Another common use case is I want to take a consistent backup/copy. For large databases, this is a very, very long snapshot to maintain, if MVCC, or a lot of locks to acquire, if 2PL. How does the system act then?

All in all, I think it's pretty neat, and I like that someone is dealing with OLTP database problems with a mind towards easing one's burden via transactions.

haberman3• 14 years ago

> Okay. I open a transaction for three days (let's say indefinitely), maybe reading a few tuples here and there and updating a few, but never committing. How does this system not fall over?

They say they use optimistic concurrency control, so your writers aren't taking any locks, instead the transaction will just fail when you commit if any of your read/modify/write's were modified in the meantime.

fdr6• 14 years ago

Ah. So that probably means a consistent copy or backup is probably practically impossible.

Andys• 14 years ago

Re your first point - this is a problem for PostgreSQL also.

jeltz• 14 years ago

I would say you should not do that in almost any database. Do not never keep transactions open for much longer than necessary.

fdr6• 14 years ago

Sure it is. But I would say, more to the point, it's a problem for systems that support consistent snapshots. The question is: how does it deal with the failure condition?

voidmain9• 14 years ago

FoundationDB co-founder here.

As other posters have already guessed, we don't support long-running transactions (we only keep multi-version information for a short time). So if you keep a transaction open for a long time it will not commit (after a while reads will start to fail, too).

It's not architecturally impossible for us to support long-running read snapshots, but it is expensive for our storage servers to keep snapshots alive for a long time. So our backup solution instead works by backing up transaction logs while doing a non-isolated read of the database. At restore time we will replay the logs to get back a consistent point-in-time snapshot.

The reason that we see higher burst than steady state performance has nothing to do with transactions. We have to do reads from SSD as the application requests them, and we have to make writes durable by logging them right away, but as you surmise we can defer the hard work of doing random writes to our btrees for a while. Even a workload that is 90% reads benefits a lot from deferring writes because writes have to be 3x replicated and because consumer SSDs are comparatively slow at mixed read/write workloads. Read only workloads will not see any initial burst (and might see a ramp-up time from cold cache effects).

fdr6• 14 years ago

> It's not architecturally impossible for us to support long-running read snapshots, but it is expensive for our storage servers to keep snapshots alive for a long time. So our backup solution instead works by backing up transaction logs while doing a non-isolated read of the database. At restore time we will replay the logs to get back a consistent point-in-time snapshot.

Sure, it's all technically possible -- Postgres will simply stop collecting garbage via VACUUM, for example -- but the results are not usually very pleasant. But it sounds like instead your solution to the snapshot-read is more like how the online binary backups work, whereby you combine transaction logs with inconsistent reads. That system works very well. On the other hand, "logical" backups have been a major problem for me; sidestepping this by using lower level transaction log mechanics and dirty reads I think is a much better idea.

> The reason that we see higher burst than steady state performance has nothing to do with transactions. We have to do reads from SSD as the application requests them, and we have to make writes durable by logging them right away, but as you surmise we can defer the hard work of doing random writes to our btrees for a while. Even a workload that is 90% reads benefits a lot from deferring writes because writes have to be 3x replicated and because consumer SSDs are comparatively slow at mixed read/write workloads. Read only workloads will not see any initial burst (and might see a ramp-up time from cold cache effects).

I see. It's not read-only, it's 90/10, I misread. That makes more sense. What's your read-only performance? Presumably it would be impacted by requiring fencing to deal with cases involving network partitions, which is one of the problem with non-partitioned and consistent systems systems, as far as I can tell: reads need to check for liveness, and that's much harder than....doing nothing.

cbsmith4• 14 years ago

Honestly... if you have figured out how to scale with full ACID... I have a hard time believing that having SQL support in there is going to "kill" your ability to scale.

Roboprog• 14 years ago

I'm guessing that no-SQL documents are written at a more course granularity than SQL table rows. One document could well represent 3, 4, 5 rows in a normalized SQL database.

OTOH, if you put all your "documents" in 3rd normal form, you might not see much gain.

cbsmith4• 14 years ago

The overhead with transactions doesn't come from the fields being in different tables...

cbsmith4• 14 years ago

Looking at what they are providing, it's a shared nothing architecture, with data stored in ordered form. Technically, you can make this work in a very scalable fashion with an SSTable style store.

Think of it as LevelDB with a distributed B+ tree (or even just a few extra levels) handling the partitioning between nodes. That can scale quite well, and wrap updates and reads with snapshots at very low overhead to provide all the key bits to handle ACID in a distributed database.

shin_lao4• 14 years ago

What's problematic is when you have a transaction that span amongst several nodes and you need to reconcile several transactions (commit phase).

The usual way to do it is to refuse the transaction if a conflict is detected, however this is a real performance and useability issue for a NoSQL database.

What I'm implying is that if you have ACID transactions but they can fail very easily, you don't offer much...

cbsmith4• 14 years ago

SSTable style databases are very cheap to snapshot, so you could totally handle this MVCC style. I buy the logic could get nutty in some pathological cases, but performance wise it shouldn't be much of a problem.

jynnan3• 14 years ago

It's not that hard to retry a transaction until some deadline is reached. Many programming languages have standard library facilities to make this extremely easy.

If the database is fast enough and transaction failure is at least moderately unlikely, it's not really an issue.

damian2000• 14 years ago

I recommend having a read of this article explaining the differences between CAP and ACID: http://www.infoq.com/articles/cap-twelve-years-later-how-the...

TL;DR

ACID represents one design philosophy at the consistency end of the consistency-availability spectrum. According to CAP, to have the maximum Consistency of ACID, you'd need to trade-off against a lower Availability and/or Partitionability.

jynnan3• 14 years ago

"Trading off against partinonability" is kind of a strange concept -- not only can you not prevent network partitions, you usually can't tell when they've happened until it's too late.

The real upshot of the CAP theorem is that you have to choose what goes in the face of a partition. An ACID system says you lose availability (writes, and possibly reads, fail if you can't reach a majority of the participants); an Eventually Consistent system may pick either.

haberman3• 14 years ago

This is a beautiful site with tons of information, graphs, lists, examples, etc. and yet omits the one thing that underpins it all: how it actually works.

The claims are: serializable ACID cross-machine transactions "without performance penalty." MVCC with optimistic concurrency control.

Optimistic concurrency control means that the server has to check the version on all modified data before the transaction commits. Cross-machine transactions mean that this version check has to happen on multiple machines. ACID means that both machines have to either commit (if all the version checks succeed) or roll back. How are you going to reconcile all of these requirements without resorting to two-phase commit, which most certainly has a performance penalty? And how are you going to get serializable transactions across machines without processing these cross-machine commits one at a time (waiting for two-phase commit each time)?

I'm just not seeing it.

aphyr3• 14 years ago

Serialized transactions over a broad distribution of keys isn't a huge problem, but you're right: this bodes poorly for hot data. I'm more concerned about their CAP semantics: I'm sorry, but claiming multidc availability and acid transactions is not gonna work.

voidmain9• 14 years ago

FoundationDB co-founder here.

As any ACID database must, we choose Consistency over Availability in the CAP theorem sense. This means that when a datacenter is disconnected or poorly connected to the internet, the database is not available for writes in that datacenter. If you are serving a web site to the internet, and need it to stay available if one Amazon AZ fails, this isn't too bad.

Some applications really do need disconnected operation, and thus 'AP' semantics. (For example, a grocery list for a mobile phone, or an application that must be available everywhere if the entire Internet partitions.)

voidmain9• 14 years ago

FoundationDB co-founder here.

I'll try to address your questions. This isn't a complete explanation of how our system works!

Most people trying to solve this problem have approached it by trying to take multiple machines, each capable of processing transactions, and do distributed transactions between them. This leads to systems that do local transactions fast but global transactions much more slowly, and a tradeoff between the quality of fault tolerance (paxos=good, 2-phase=bad) and commit performance.

Our approach is instead to decouple the first job you mention (checking if transactions conflict to provide isolation) from the second job (making transactions durable). These can then be scaled independently.

The first job, conflict resolution, is a small fraction of the total work and needs access only to the key ranges used by each transaction, not the existing contents of the database or the actual values being written. This allows it to be very fast.

The rest of the system is a consistent, durable distributed store that has to preserve a predetermined ordering of writes, and provide a limited window of MVCC snapshots for reads.

Does that help at all?

bobobjorn• 14 years ago

That sounds alot like partialy atleast, the same model datomic has?

haberman3• 14 years ago

I still don't see it. Say you have conflicting transactions T1 and T2 that both update data on two machines M1 and M2. If these transactions race to commit, how do you guarantee consistency? Even if you have a perfect, zero-latency oracle that can tell you that T1 and T2 conflict, you still need M1 and M2 to form consensus about which transaction commits first, and to make sure that both machines either commit or roll back (I am assuming that both machines have their own transaction log). It still sounds to me like 2PC is required.

Specific questions: does each machine have its own commit log? Is each machine authoritative for its range of the keyspace, in that it can serve reads without having to consult other machines?

d0ugal3• 14 years ago

What exactly is "YesACID" ?

ericcholis• 14 years ago

Perhaps I'm being a designer-snob, but Twitter's Bootstrap has quickly become too popular. I honestly have a hard time taking a site seriously when it uses the default Bootstrap look.

Having a unique visual identity is extremely important.

Kudos• 14 years ago

I don't think you're being a designer-snob. Bootstrapping your design with Twitter Bootstrap is cool, making Twitter Bootstrap your design is not.

Wilya• 14 years ago

Honestly, if someone wants to sell me a database, I'd prefer they spend money on database engineers rather than on designers. To each his own, I guess.

forgotAgain• 14 years ago

I don't think it's a designer-snob thing. Taking the bootstrap provided website and a lot of overreaching claims of db goodness gives me an immediate sense of there being no there, there.

crazygringo• 14 years ago

It's true. It's fine for a small MVP in beta, and even a good sign (shows they're concentrating on the product details the are important).

But launching an entire new database, does deserve its own visual identity. That said, their logo is rather nice!

jhull• 14 years ago

Is it just me or is there a new "superior-than-the-others" NoSQL solution every week?

bsg75• 14 years ago

Today's database is better than yesterday's.

wmf4• 14 years ago

Hey guys, I like units on the Y axis of graphs.

Dave_Rosenthal2• 14 years ago

So do I. Fixed :)

christianaranda• 14 years ago

I don't need to be convinced. No doubt this works as advertised. Nicely done.

d0ugal3• 14 years ago

Switching all my production code to use their alpha as we speak.

js4all2• 14 years ago

The claim to be the first acid compliant NoSQL database is wrong.

Bigcouch and Couchbase are around for a while. There is also CouchDB (without the scaling).

voidmain9• 14 years ago

I was nervous writing that sentence, because it's hard to be sure of the truth of any claim to be 'first'! But none of the examples you mention, to my knowledge, provide multi-key ACID transactions. A compare and set facility for an individual key/document/row/entity group/etc is a very useful feature, but cannot be used to provide atomicity or isolation for transactions that read or write more than one key.

We plan to write something longer talking about the different levels of support that various products provide for A,C,I, and D.

js4all2• 14 years ago

> But none of the examples you mention, to my knowledge, provide multi-key ACID transactions

If you see it that way, you are right. The guaranty is just for one key/document pair (the key can be a key vector though). There is no way to have a commit over several key/document pairs with these databases.

If FoundationDB can do that, it is a big plus. If it can do it in a cluster, your product will have a great future.

jynnan3• 14 years ago

Your performance tests seem to have been run on a cluster that lives on one gigabit LAN. Have you tested a cluster that's distributed geographically?

FoundationDB: A new generation of NoSQL database

Related Stories