petermattis

Cgo exists because goroutine stacks are small and grow on demand. C/C++ expect large fixed size stacks. While you can technically call into C++ without going through cgo (see https://github.com/petermattis/fastcgo), you can't avoid the stack problem. Cgo solves it by switch from the goroutine stack to the stack of the thread underlying the goroutine.

Direct IO is on our radar, though when I experimented with enabling direct IO in RocksDB it only hurt CockroachDB benchmarks. This was several years ago. I believe newer releases of RocksDB have made improvements in this area.

Is Pebble/CockroachDB capable of saturating the backplane with requests in parallel?

Yes.

Does it multiplex a single query by dispatching smaller requests to a thread-pool?

Yes, though it depends on the query. Trivial queries (i.e. single-row lookups) are executed serially as that is the fastest way to execute them. Complex queries are decomposed along data boundaries and the query parts are executed in parallel next to where the data is located.

Pebble and FoundationDB are apples and oranges. Pebble is per-node KV storage engine. FoundationDB is a distributed multi-modal database. Internally, FoundationDB uses a library like Pebble for the per-node data storage. I think at one point it used SQLite. I'm not up to date on what it currently use. I seem to recall FoundationDB was writing their own btree-based node-level storage engine to replace the usage of SQLite.

The equivalent of FoundationDB is present inside of CockroachDB: a distributed, replicated, transactional, KV layer. This is where a big chunk of CockroachDB value resides. This is where our use of Raft resides. Pebble lies underneath this.

Yes. So far performance was worse in experiments, and the durability improvements are questionable because it is extremely difficult to get a clear understanding of the durability semantics of direct IO. If you can find a pointer to clear documentation of what those semantics are I'd be extremely interested in reading it.

Damned for using a unique name (CockroachDB), and damned for using an innocuous one.

PS PebblesDB was a research project and is dead as far as I know.

Pebble does not currently implement MultiGet as CockroachDB did not use RocksDB's MultiGet operation. CockroachDB can use multiple nodes to process a query by decomposing SQL queries along data boundaries and shipping the query parts to be executed next to the data. CockroachDB can't directly use MultiGet because that API was not compatible with how CockroachDB reads keys.

RocksDB MultiGet is interesting. Parallelism is achieved by using new IO interfaces (io_uring), not by using threads. That approach seems right to me. See https://github.com/facebook/rocksdb/wiki/MultiGet-Performanc.... My understanding is that io_uring support is still a work in progress. We experimented at one point with using goroutines in Pebble to parallelize lookups, but doing so was strictly worse for performance. Experimenting with io_uring is something we'd like to do.

As far as I'm aware, the fsync/fdatasync sharp edges are around what happens after an fsync/fdatasync failure. My understanding is that you can't rely on anything. The only sane option is to crash the process and attempt recovery on restart. Even that is fraught because data can be in the OS cache but not synced to disk. Pebble (and RocksDB) both take a fairly pessimistic view of what can be recovered. Sstables that were in the process of being written are discarded. The WAL an MANIFEST (which lists the current sstables) are truncated at the first sign of data corruption. Getting all of this right definitely takes time and effort.

From the Rebello paper: > However, on restart, since the log entry is in the page cache, LevelDB includes it while creating an SSTable from the log file.

Pebble and RocksDB both inherited this behavior. The nuance here is that the sstable is then synced to disk and no reads are served until the sync is successful. If the machine were to crash before the sstable was synced, upon restart we'd rollback to the durable prefix of the log.

So I understand the rationale for writing your own storage layer and think this is an awesome project, but there's something missing for me. One of the issues Peter brings up is they've come across a number of serious bugs in RocksDB. My question is, why would Pebble have less bugs. In fact, I would expect it to have significantly more bugs because Cockroach is the only company using Pebble.

We're only worried about functionality in Pebble used by CockroachDB. RocksDB has a huge number of features that sometimes have bugs due to subtle interactions. There is a very stable subset of RocksDB: the configuration and specific API usage patterns used internally by Facebook. That precise combination has seen extreme testing. But that isn't the subset of RocksDB used by CockroachDB. I would guess that the most significant testing of the subset of RocksDB used by CockroachDB is the testing we do at Cockroach Labs. Now that testing is being directed at Pebble along with the Pebble-specific testing detailed in the post.

For example, it's possible the filesystem had synced some of the buffered data to disk, but not all of it. There's no guarantee about what buffered data was synced to disk. All you know is that some, all, or none of it made it to disk.

The filesystem does provide guarantees when you use fsync() and fdatasync(). Postgres relies on these guarantees. So does RocksDB. Pebble's usage of fsync/fdatasync mirrors RocksDB's. Our crash testing is not testing the filesystem guarantees, only that we're correctly using fsync/fdatasync (which is hard enough to get right).

The TLDR is that the GC did cause problems so we had to avoid it for the block cache. Luckily we were able to do so without exposing the complexity in the API. Not for the faint of heart. Don't try this at home kids.

Performance is "ok". CockroachDB 1.0 won't be winning benchmarks, but that was never the goal. We have seen near linear scaling from 3-64 nodes on a uniformly random write workload. And the new distributed SQL execution engine has shown O(num-nodes-in-cluster) speedup on some queries. But none of this has been wrapped up into published numbers, so take it with a grain of salt.

(Cockroach Labs co-founder here)

This is a really hard question to answer because there are so many different scenarios to test. Even if you restrict the question to KV operations a simple answer isn't forthcoming. For example, if you want a distributed system with the fastest writes, then Cassandra is your best choice. Oh, but you better pay a lot of attention to all of the caveats regarding tombstones and node outages and overwriting data.

Overall, performance has improved a lot over the past year, but we still have a lot of work to do. Some of the high-contention scenarios that caused poor performance in the Jepsen tests have seen a huge improvement (10-100x), though admittedly we were starting from a poor position.

So far we've performed the most significant comparisons against Cassandra and MongoDB. In terms of performance Cassandra > MongoDB > CockroachDB, but note that most (all?) benchmarks test idealized situations. YCSB tests ideal write scenarios where data is never overwritten or deleted and data correctness is easily achieved because there are no node outages.

Unsurprisingly, PostgreSQL will outperform a single-node Cockroach cluster for simple operations. The internal target was to be within 2x of PostgreSQL performance for simple INSERT, UPDATE, DELETE and SELECT statements. We were at that target earlier in the year, but a quick check reveals that we've slipped a bit.

Sorry I don't have anything more concrete to offer at this time. I realize I've been saying "we're working on it" for too long. Really, we are.

(Cockroach Labs co-founder here)

CockroachDB also stores SQL metadata inside of the KV store, but that metadata is also gossiped around the cluster (i.e. it is replicated to every node) so that SQL execution almost never has to read it from the KV store. Handling changes to this SQL metadata is challenging and required a design [1] that is unable to take advantage of the easy correctness of simply doing reads from the KV store on every operation.

[1] https://github.com/cockroachdb/cockroach/blob/master/docs/RF...

Hi Chris, we describe CockroachDB as a SQL database because that is what we're aspiring to. The missing functionality (i.e. joins) is on our near-term roadmap.

(employee/founder here)

Re joins: we have top men working on it right now. Joins are definitely an interesting problem for a distributed database.