andreimatei1

Side-Eye is massively inspired by DTrace in some of its raw capabilities and the basic idea of dynamic instrumentation. Beyond that, they're very different. At a low level, DTrace is primarily geared towards debugging the kernel, whereas Side-Eye is about userspace. DTrace's support for the DWARF debug information format used on linux is limited. The interaction model is different - for DTrace you write scripts to collect and process data. DTrace works at the level of one machine, whereas Side-Eye monitors processes across a fleet. In Side-Eye you interact with a web application and you collect data into a SQL database that you can analyze. Side-Eye is also a cloud service that your whole team is supposed to use together over time.

And then there are more technically superficial, but crucial, aspects related to specific programming language support. Side-Eye understands Go maps and such, and the Go runtime. It can do stuff like enumerate all the goroutines and give you a snapshot of all their stacks. We're also working on integrating with the Go execution traces collected my the Go scheduler, etc.

Print debugging is the tool most people reach for when they can, but its biggest problem is that you have to change the source code to add the printfs. This is impractical in many circumstances; it generally only works on your local machine. In particular, you can't do that in production environments, and that's where the most interesting debugging happens. Similarly, traditional debuggers are not available in production either for a lot of modern a software -- you can't really attach gdb to your distributed service, for many reasons.

What print debugging and debuggers have in common, in contrast to other tools, is that they can extract data specific to your program (e.g values of variables and data structures) that your program was not instrumented to export. It's really a shame that we generally don't have this capability for production software running at scale.

That's why I'm working on Side-Eye [1], a debugger that does work in production. With Side-Eye, you can do something analogous to print debugging, but without changing code or restarting anything. It uses a combination of debug information and dynamic instrumentation.

[1] https://side-eye.io/

This answer should go to the top.

Very nice work! It seems that Delve is used for inspiration and development, but Golang code is always compiled with frame pointers, I think. Is that right?

Global Tables let database clients in any region read strongly consistent data with region-local latencies. They’re an important piece of the multi-region puzzle — providing latency characteristics that are well suited for read-mostly, non-localized data.

Cockroach lets you hash-shard your keys n-ways if you choose to through a feature called hash-sharded indexes (https://www.cockroachlabs.com/docs/stable/hash-sharded-index...) That's a trade-off: costlier range scans (a range scan turns into n scans) for avoiding a throughput-limiting hot spot.

its lack of ACID guarantees

Our transactions implementation is our crown jewel. You might want to check your sources.

I think the "default" will evolve with whatever offers the best "serverless" experience in the public clouds. In particular, the cheapest and most granularly-billed option.

I agree that right now it doesn't make sense.

This CRDB engineer respectfully disagrees. This thread takes it as a given that a non-distributed DB is better if you don't need to scale up (i.e. if you run a single "node"). Let me offer a couterpoint: it's easier to embed CRDB into some software you're distributing than it is to embed Postgres. This is to say, we do try to compete at every scale (well, perhaps not at SQLite scale). CRDB doesn't have all the SQL features of PG, but it does have its pluses: CRDB does online schema changes, it is generally simpler to run, comes with a UI, comes with more observability, can backup to the cloud, can be more easily embedded into tests.

Online schema changes are a big deal; the other thing that I hope will help us win small-scale hearts and minds is the ever-improving observability story. I hope CRDB will develop world-class capabilities here. Other open-source databases traditionally have not had many capabilities out of the box.

Working on it.

Would you mind providing a link to the empty arrays patch? Is this something CRDB should fix?

At GitHub they should be using CockroachDB (:P), a scalable relational database that also supports schema changes. Then their FKs would work just fine (albeit with some performance implications).

Perhaps I could emphasize things differently. FWIW, besides the fact that having a foreign key constraint in that schema would prevent the badness from happening, the even bigger reason why scenarios like that are unlikely is that, realistically, for Tobi to reply to a comment, he must have seen that comment he's about to reply to, and it's very hard to imagine a scenario where he'd see it but still have the response transaction not read it (cause if the txn read it, the two transactions wouldn't be independent any more and so they'd be well ordered). The foreign key is just one way of ensuring that the read happens.

The suggestion that this anomaly can occur in FaunaDB is wrong:

You're right. I've removed the reference to FaunaDB. Sorry about that. What I've meant to reference is the snapshot isolation of FaunaDB's read-only transactions, but the comment was in the wrong context for that.

Hi Gab_CV, this is really cool! Congrats! Are you planning on writing any notes about what the port took? You commits are quite clean, but still some narrative would be really interesting.

We have clients using CRDB in pretty constrained environments, and they use it primarily because of the easy administration. I think you'll find it easier to use than a MySQL or Postgres, for example.

https://github.com/tbg/bikesheddb

Postgres supports different modes of replication, including synchronous replication to avoid losing any data on node failure.

I would like to understand this better, for my edification. The best source on the exact semantics of pg replication that I've found were in this talk by Andres Freund: https://www.youtube.com/watch?v=VkzNL-uzBvA

What I understand is that "synchronous replication" means that the master commits locally, then waits for a slave (or k of them with quorum replication) to receive (and possibly also to apply) a commit before acknowledging the commit to the client. So what does that mean for "avoiding losing any data on node failure"? Commits acked to their client are indeed durable. But what about commits that are not acked? For example, is it possible for the master to apply a commit which is then seen by another query (from another client), then crash before the commit is sent to the slave? And so, if we then failover to the slave, the commit is gone (even though it had previously been observed)? This would qualify as "losing data".

Similarly, in that talk, Andres mentions that there is a window during which "split brain" is possible between an old master and a new master: when doing a failover, there's nothing preventing the old master from still accepting writes; so you need to tell all the clients to failover at the same time - which may be a tall order for the network partition scenarios when these things become interesting. If the old master is still accepting writes, then these writes diverge from the new master (so, again, lost data). With sync replication, I guess the window for these writes is limited since none of them will be acked to its client (but still, they're visible to other clients) - the window would be one write per connection.

I'm also curious to understand better how people in the Postgres world generally think about failoverd, both in vanilla pg and in Citus). I generally am able to find little information on best practices and the correctness of failovers, even though they're really half the story when talking about replication. For example, when replicating to multiple slaves (even with quorum replication), how does one choose what slave to failover to when a failover is needed? What kind of tool or system decides which one of the slaves has the full log? My intuition says that this is a pretty fundamental different compared to a consensus-driven architecture like CockroachDB: in Cockroach the "failovers" are properly serialized with all the other writes, so this kind of question about whether one particular node has all the writes or not at failover time is moot.

Ozgun has done a good job describing some of Citus' benefits; allow me to write the CockroachDB-biased answer. Btw, congrats to all the Citus people on your launch.

The highest-level architectural difference is, in my opinion, the fact that CockroachDB aims to not trade any features of single-node SQL databases for the distributed nature of our clusters. In contrast to Citus, you get ACID (serializable) transactions regardless of how the data ends up being distributed - so touching more than one shard/node in a transaction does not weaken any consistency guarantees. Similarly, DML statements can modify distributed data with no fuss. The sharding is transparent in crdb; one doesn't need to explicitly declare any partitioning on a table or choose a shard count. We do give you, however, control over the data locality should you need it.

The other big architectural difference has to do with our availability/durability guarantees. In crdb, all data is replicated, and the replication is done through a consensus protocol. This means that data appears to be written at once on multiple replicas. Losing any one replica at any point in time is not a big deal; all the data that had previously been written is still accessible. You'll never lose any writes because of a machine failure. This is in contrast to most master-slave type architectures, where generally data is not "atomically" written across the master and the slave. Whenever a slave is promoted to be a master, you will probably incur some data loss. It is my understanding that Citus falls in a flavor of such a master-slave architecture.

Now, this all doesn't speak about the current blog post in particular; crdb does not currently have a managed service offering.

I believe this suggestion came up at the time. I think we didn't do anything like it because it would have been more complicated both to implement and to describe, and because it's not particularly common for people to use/need the values that can be returned by the statements that we can parallelize. The future, though, is long and open.

Why do you say that they don't hold, exactly? We take our consistency guarantees, and correctness more generally, deadly serious. I think it's fair to say that we did quite well on Aphyr's review; he's found some quite obscure issues that were quickly fixed. More details here: https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j...

I can all but promise you that no CRDB dev will tell you to go F* yourself :P

I'm not sure how interleaved tables help you with multi-tenancy, but I'd be curious to find out. If, by multi-tenancy, you mean sharing a CRDB cluster between different applications or different instances of the same application, or different organizations all-together (say, in a hosted cluster that 3rd parties get access to), we don't have particular support for any of these. If each client is configurable with a user and a database, you can created different users/databases for each one and set the appropriate permissions. You can even set different placement options for the different databases if you want some in Asia and some in Africa. When it comes to usage accounting, resource isolation, specific monitoring, there's nothing built-in at the moment.

FWIW, CRDB should be a good fit for people worried about database lock-in. You'd only be "locked in" if you discover that we're much better than the alternatives :) CRDB uses the standard SQL language. Also, CRDB speaks the PostgreSQL wire protocol; for using CRDB you'd be using the Postgres drivers for different languages. So, more or less, any application that can work with us should also work with Postgres (and vice versa). Moreover, the Postgres drivers generally respect other higher interfaces (odbc, jdbc) so, more or less, a CRDB application should also work with any other established SQL database (and vice versa). CRDB can also export your data in formats easily importable by other SQL databases.

Besides the blog, you can find interesting stuff in the source repo, in the technotes and RFC dirs: https://github.com/cockroachdb/cockroach/tree/master/docs/te... https://github.com/cockroachdb/cockroach/tree/master/docs/RF...

There's also a bunch of technical tech-talks linked from our company's website: https://www.cockroachlabs.com/community/tech-talks/

Well, those numbers, if I remember correctly, refer to insert rate in the face of some extreme amounts of contention - tons of overlapping reads and writes (that's how the test was trying to trigger consistency violations). And moreover, they were measured while the Jepsen test framework was messing with the cluster. Contention is a problem for every database, and particularly so for CRDB. In the absence of contention, we routinely see thousands of queries per second per node.

Since the time of that analyses, we have done a significant amount of work for speeding up these high-contention scenarios, with quite dramatic differences in some cases we looked at. We pretty much changed our transaction execution model from a more "optimistic" one where transactions can abort each other and induce thrashing, to something resembling more the traditional row locks. So, hopefully, even for these atypical uses cases we should generally perform much better.

Anyway, show me a database that passes Jepsen and I'll show you a database that only accepts writes to a single node.

I know one - CockroachDB :). Relational, strongly-consistent, SQL, elastic, multi-active replication - the dream! (I work on it)

https://www.cockroachlabs.com/blog/cockroachdb-beta-passes-j... http://jepsen.io/analyses/cockroachdb-beta-20160829