sergei

Several problems here:

1. Unlike your dataset, the tpcc dataset for the benchmark was not memory resident. Total dataset size was 786G. Just shrinking the dataset to fit in memory would substantially change the numbers.

2. The tpcc workload is much more complex than your benchmark. It doesn't make sense to compare a tpcc transaction, of which there are multiple flavors, to your workload.

3. All Clustrix write transactions are multi-node transactions. There's a 2x data redundancy in the test. We do not have a master-slave model for data distribution. Docs on our data distribution model: http://docs.clustrix.com/display/CLXDOC/Data+Distribution.

-----

Now we've also done tests where the workload is a simple point updates and select on memory resident datasets. For those kinds of workloads, a 20 node cluster can do over 1M TPS.

What do you base your performance claim vs Clustrix on?

I frequently see apps with thousands of distinct query signatures. Having a developer manually chose indexes, join ordering, aggregation method, (just to name a few) for every single query, and then select multiple plans because input parameters absolutely do result in scenario where plans can be 10,000x off in performance -- well, that's just untenable.

Not trying to claim that CBOs are the panacea here, but let's be realistic. Having developers manually plan every single query is not the right choice.

In practice, you end up with apps which dynamically build up predicates in different sections of the code. And when you combine that with many predicates, many tables, and other constraints such as ordering or aggregates, things get complex pretty quick.

Even if you have the best developers who understand all the in an outs of the dataset, re-implementing an optimizer in the app is rarely the right choice.

A decent cost based optimizer will be right most of the time, and you have hints for the cases where it falters.

Your approach is simply hiding the query optimizer choices inside of the app, making it more fragile.

Let's say I have two fields: a and b, both of which have an index. And then I have a query with predicates across both a and b. For example, a = 10 and b = 20.

One of the core facilities within an optimizer is selectivity estimation. By looking at the statistics, the optimizer will see that a = 10 might look at 10,000 rows while b = 20 might only look at 10. So the optimal and desired choice would be to use index over b.

However, the exact same query construct with different parameters (e.g. a = 50 and b = 3) might flip the index selection.

Now let's imagine I had to implement this inside of my app. Every time I have such a constraint, the app has to become aware of selectivity to know which indexes to use based on input parameters.

Check out a previous comment thread on the subject: https://news.ycombinator.com/item?id=5068489

Check out a previous comment thread on the subject:

https://news.ycombinator.com/item?id=5068489

Yes, but they are in-memory only. I think the largest database they support is 50GB.

It's the kind of solution that works really well if you have key-value store problem where you want really low latencies. Telco call session state and session state for gaming comes to mind.

But beyond those use cases, you start running into a lot of architectural limitations...

1) Both joins and aggregates are processed in parallel. In short, the larger your cluster, the faster we evaluate a complex query because we can bring more hardware to the problem.

You can get a better idea of how we scale distributed joins from one of my blog posts:

http://www.clustrix.com/blog/bid/242790/Scaling-Distributed-...

Once we complete the join, we will also perform the aggregate SUM operation across multiple nodes, getting individual sums, and then forwarding the results to another node for a final tally.

As a rule of thumb, each node has the throughput of about 70% of MySQL on the same hardware. So if you have a complex query that can take advantage of the cluster (100k rows per table examined qualifies), then you will see a 10x speedup on a 16 node cluster over a single instance.

However, that assumes that your entire data set fits in cache on the single instance box. As soon as that's no longer true, it's more likely that you will see a 100x or more speedup from Clustrix because you will have way more memory available for the buffer cache.

For example, if you have a 32GB single instance MySQL server, but you have more than 64GB of working set size, you're going to be limited by IO.

On Clustrix, if you move this to say a 4 node cluster, you are going to have an effective cache footprint of 128GB. So Clustrix will operate entirely out of cache.

2) We're fully ACID compliant using a two phase commit w/ PAXOS for transaction resolution. Because we're MVCC, readers never have to take out locks. To coordinate WRITE access to the same data, we have a row level distributed lock manager. That means that the entire cluster participates in lock management.

So yes, this has a higher communication overhead over a single server. If you have a small write workload with very little concurrency, you will see better query latencies from a single instance database.

However, as soon as you start getting to more serious write workloads with higher concurrency, Clustrix will distribute your writes across several machines in the cluster. We can handle an aggregate write throughput you can't touch with a single instance database, even with a fancy/expensive storage array.

One of our customers, for example, has a peak throughput of 70,000 transactions/sec. They're about 90/10 read/write, so that breaks down to 10,000 TPS for writes. And they are really fond of multi-row updates, insert .. ignore, insert .. on dup update kind of queries. So it's not exactly a lightweight key-value application. Prior to Clustrix, their MySQL installation peaked out at 5,500 TPS total (read+write). They now have a 15 node Clustrix cluster.

It's true, you can always construct a workload that will not distribute well. But these are generally rare degenerate cases that stem from poor application design, and would perform equally poorly on a single instance database.

It's a good question. We built the entire database from the ground up, without any MySQL code. So while we support most of the frequently used features, we don't support everything.

You can get a list of unsupported features and differences from our docs:

http://docs.clustrix.com/display/CLXDOC/Unsupported+Features

http://docs.clustrix.com/display/CLXDOC/Feature+Differences

Having said that, our customers find that the value that they get out of scale out, fault tolerance, and performance is worth the tradeoffs of not supporting every single MySQL feature.

Honestly, our goal is not to be 100% MySQL compatible. We're aiming to be compatible enough to capture a significant share of existing MySQL installs, but we're really after building a truly scalable relational db.

Try it. You'll like it :)

It's different from a typical sharding approach (including what MongoDB does). In their model, you take a single key and distribute your data using that key (e.g. user_id). The problem surfaces when you look at secondary indexes.

If you have a secondary index say on user_location, and you want to query by that index, you don't know which shard to go to. So you end up broadcasting.

Another problem is enforcing unique index constraints.

With Clustrix, every table and index gets its own distribution.

So if you have a schema like this:

foo(a, b, c, d) unique idx1(b,c) idx2(d)

Clustrix treats each table and index as a different distribution. So if I need to look something up by d, I know exactly which node has the data. I can also enforce index uniqueness.

Some reading material on the distributed approach:

http://www.clustrix.com/blog/bid/242790/Scaling-Distributed-...

http://www.clustrix.com/Portals/146389/docs/clustrix_a_new_a...

San Francisco, CA Clustrix: Systems Developers

Clustrix has developed a highly scalable distributed database system from the ground up. We are looking for skilled systems developers to help us with the next generation of Clustrix Database.

As a candidate, you should be an experienced C developer and proficient in concurrent and asynchronous system principles.

Additionally, experience in any of the following areas is highly preferred. It's a sample of the kinds of problems Clustrix developers are faced with on a daily basis:

* Compiler design and implementation * Distributed query planning and optimization * Distributed concurrency control mechanisms * Fault tolerance in distributed systems * Distributed transaction management

jobs@clustrix.com

The problem with Mongo's concurrency approach is that when you introduce writes to the equation (which take out an exclusive lock on the database), the system becomes unusable.

Here's a link to some of the benchmarking work I did. Ignoring the comparison to Clustrix, you can clearly see readers getting starved out when a writer does io.

http://sergeitsar.blogspot.com/2011/01/mongodb-vs-clustrix-c...

Sorry, but "hardware failure" is a fault, and when you can't deal with it, you're not tolerant. And with larger clusters, you see hardware faults on a regular basis. So saying we're ok in the nominal mode is not fault tolerance.

1. Yes, I recognize that MongoDB will automatically fail over when we go from N nodes in the set to N - 1. But how do I get back to N nodes? That's completely manual.

2. What happens when I read an update that succeeded on the master but then later fails on the slaves?

1. Say I have a 2 node replica set. Now a replica dies, permanently. How is the recovery automated? These are quotes directly from your docs:

http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+R...

"1. Delete all data. If you stop the failed mongod, delete all data, and restart it, it will automatically resynchronize itself. Of course this may be slow if the database is huge or the network slow.

2. Copy data from another member. You can copy all the data files from another member of the set IF you have a snapshot of that member's data file's. This can be done in a number of ways. The simplest is to stop mongod on the source member, copy all its files, and then restart mongod on both nodes. The Mongo fsync and lock feature is another way to achieve this. On a slow network, snapshotting all the datafiles from another (inactive) member to a gziped tarball is a good solution. Also similar strategies work well when using SANs and services such as Amazon Elastic Block Service snapshots.

http://www.mongodb.org/display/DOCS/fsync+Command "Lock, Snapshot and Unlock

The fsync command supports a lock option that allows one to safely snapshot the database's datafiles. While locked, all write operations are blocked, although read operations are still allowed. After snapshotting, use the unlock command to unlock the database and allow locks again

2. Really? Is this wrong then?

http://www.mongodb.org/display/DOCS/Replica+Set+Design+Conce...

"Writes which are committed at the primary of the set may be visible before the true cluster-wide commit has occurred. Thus we have "READ UNCOMMITTED" read semantics. These more relaxed read semantics make theoretically achievable performance and availability higher (for example we never have an object locked in the server where the locking is dependent on network performance).

The article is not about the DBMS interface. It's about Fault Tolerance (what happens when stuff breaks) and Availability (can I still use my database when there is a fault).

MongoDB claims to support both. So does Clustrix. I'm comparing both claims.

A large part of the message I'm trying to convey is:

1. A DBMS is much more than just the interface. I'm going to write more on the subject. Whether you're using SQL, datalog, BSON, etc. -- there's a broader set of desirable features that's more important than the query language.

2. I'm not saying I agree with what the NoSQL folks are saying, or their justifications. Let's be honest: there's a prevalent sentiment that relational SQL based databases do not scale, and even further, that they somehow cannot scale. That's just not true.

I saw a video the other day of some guy at Google giving a talk about the database behind the app engine. At one point someone in the audience asked about scale and SQL. His response was "Well, how well does SQL scale?" Everyone in the room laughed.

I updated the post with the benchmark source.

OK. Fair enough. I'll post the benchmarks.

I chose Mongo because it gets a lot more attention on HN than any other database. I don't remember the last time I saw a post on Cassandra on here...

These folks disagree with you. There are many more behind them.

http://gigaom.com/cloud/clustrix-lifts-the-curtain-on-early-...

And plenty of folks use MySQL (and PogreSQL to a much lesser extent). You just can't scale those.

And we're always looking to hire good engineers with skills in distributed systems development.

http://www.clustrix.com/about/careers/

As far as I know, no has a system like ours that can actually scale OLTP workloads. Warehousing/Analytics databases are very different beast. Can you point me at an OLTP system that I can expand without taking any loss of availability? How about supporting online schema changes?

Infiniband ports cost less than 10GE right now. There's really nothing that ties our architecture to IB. It's all about price/performance. When a better/cheaper technology becomes more available, we'll switch to that.

The database has a transitionally consistent memory cache. On a 5 node system, you have ~150GB of effective cache. And you can always add more nodes.

When you put a cache in front of the database, you sign up to manage your own cache coherence and consistency.

Are my users reading stale data? That write went into the cache, did it make it to the database? I use interface X to speak to the cache but interface Y to speak to the database. Some requests are ok to cache, some must go direct...

Or I can just get more nodes and not have to worry about it.

Absolutely. Yes to all three.

Oracle RAC allows multiple machines to synchronize access to a shared disk. Each node in a RAC cluster locks individual disk pages and brings the data to the node for processing. Only one node is working on a query a time, while synchronizing access to resources at a physical level.

Clustrix is fundamentally different. We compile the query down into query fragments and evaluate relevant portions of the query at the node which holds the relevant data. We bring the query to the data.

You can find more specifics on our architecture in our white paper:

http://www.clustrix.com/wp-content/uploads/2010/04/clustrix-...

Clustrix is all about concurrency and parallelism. While we didn't specifically focus on OLAP, our model allows us to do very well for these types of workloads because we break the query apart and let individual nodes work on their portion of the query.

While we deliver our product as an appliance, it's all based on commodity hardware.

If you had an infinitely scalable database, would you still want to put a caching layer in front of it? Most people would rather not deal with the added complexity.

In a typical sharding scheme, one would choose a partition key (typically some primary key, e.g. user id) and use that as a distribution strategy. At such granularity, hot spots are a common problem. In Clustrix, we have independent distribution/partitioning for each table/index, down to the row level. To get hot spots is if you contend on a single row -- but that's really an indication of something going terribly wrong in the data model.

ACID (transactional properties) and KV (data model) are orthogonal concepts. Our systems scales exceptionally well and supports a wide range of data models.