TiDB – a global scale distributed DB

https://pingcap.github.io/blog/2017/10/17/announcement/

by eigenrick • 9 years ago

163 39 9 years ago

pingcap.github.io

BlackjackCF• 9 years ago

Congrats to PingCAP on the 1.0 launch.

I've used TiDB for some side projects and I was really impressed. Excited to see how things continue to develop!

api• 9 years ago

Paging Jepsen. Paging Jepsen.

Edit: some work's been done:

https://pingcap.github.io/blog/2017/09/01/tidbmeetsjepsen/

anishathalye2• 9 years ago

They're also working on using a custom-built system written in Golang to test TiDB (though this is in the early stages): https://medium.com/@siddontang/use-chaos-to-test-the-distrib...

siddontang• 9 years ago

Hi @anishathalye, thanks for your https://github.com/anishathalye/porcupine. It’s a great project! Would you like to help me to check whether our model check is right or not with Porcupine? That would be highly appreciated!

foohak422• 9 years ago

So, how does it compare to cockroachdb?

So far i've seen: - Apache v2 license - Aims at compatibility with Mysql vs postgres for cockroach

shenli35144• 9 years ago

There are some key differences between TiDB and Cockroach.

1. User interface and eco-system Despite that TiDB and CockroachDB both support SQL, TiDB is compatible with MySQL protocol while Cockroach chooses PostgreSQL. You can directly connect to TiDB server with any MySQL client.

2. Architecture The whole TiDB project is logically divided into two parts: the stateless SQL layer (TiDB) and distributed storage layer (TiKV). As TiDB is built on top of TiKV, developers have the freedom to choose to use TiDB or TiKV, depending on their own business. If you only want a distributed Key-Value database, you can just use TiKV alone for higher performance and lower latency.

In a word, our system is highly-layered and modularized while CockroachDB is a P2P system. The design of our system results in the fact that we use two programming languages: Go for TiDB and Rust for TiKV to improve the storage performance.

And benefit by the highly-layered architecture, we build another project[1] to run Apache Spark to on top of TiDB/TiKV to answer the complex OLAP queries. It takes advantages of both the Spark platform and the distributed TiKV cluster.

3. Transaction model Even though CockroachDB and TiDB both support ACID transaction, TiDB uses a model introduced by Google’s Percolator. The key feature of this model is that it needs an independent timestamp allocator. Like Spanner, each transaction in TiDB will have a timestamp to isolate different transactions.

The model that CockroachDB uses is similar to the TrueTime API that Google described in its paper. However, unlike Google, CockroachDB didn’t build the atomic clocks and GPS receivers to keep the time consistent across different data centers. Instead, it uses NTP for clock synchronization, which leads to the problem of uncertain errors. To solve this problem, CockroachDB adapts the Hybrid Logical Clocks (HLC) algorithm.

4. Programming Language TiDB uses Go for the SQL layer and Rust for the storage engine layer. As Go has a Garbage Collector (GC) and runtime, we think it will cost us days to tune the performance. Therefore, we use Rust, a static language, for TiKV. Its performance is much better. CockroachDB only uses Go.

[1] Spark on TiKV: https://github.com/pingcap/tispark

foohak422• 9 years ago

Thanks for the details!

I'm pretty sure cockroach use RocksDB for the underlying storage so it's written in C++.

ansible• 9 years ago

CockroachDB uses RocksDB for the storage engine on each node, and that's written in C++ (which can be viewed as good or bad).

eloff4• 9 years ago

From their blog: "TiKV is a distributed key-value database. It is the core component of the TiDB project and is the open source implementation of Google Spanner." I think the Cockroach DB guys would take offence at that.

So how do they use a Spanner-like design without specialized hardware? I had to really dig but the answer is "We are using the Timestamp Allocator introduced in Percolator, a paper published by Google in 2006. The pros of using the Timestamp Allocator are its easy implementation and no dependency on any hardware. The disadvantage lies in that if there are multiple datacenters, especially if these DCs are geologically distributed, the latency is really high."

Using Spanner's design but without the real hardware that makes it practical seems like a step backward. A Calvin[1] based approach like FaunaDB would probably have been better - but they can't do that and still have MySQL compatibility (you can't do interactive client-server transactions with Calvin.)

I'm skeptical of a database built over a generic KV-store. There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. I can't even remember the name now when I was trying to find the post-mortem analysis for it in Google.

I'm also skeptical of a database claiming to be good at both OLAP and OLTP. One requires a column store, the other a row store. You can be half-decent as an OLAP store and good as a OLTP store. There are no column-stores that are also good at OLTP. But OLAP is the real big-data problems of today, and using half-decent for that is likely to get you in trouble. There's no reason a database can't do both by using two separate storage engines under the hood, but that doesn't seem to be the case here.

[1] https://fauna.com/blog/distributed-consistency-at-scale-span...

zzzcpan2• 9 years ago

KV-store is probably the reason why they can claim OLAP, since distributing batch load on top of it is easier.

nemothekid• 9 years ago

There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture.

The only database I can remember that did something like this was FoundationDB. They had the core layer as a K/V, and a layer above that was FSQL or similar. They got acquired by Apple, and most of their material was taken off line. I don't know if they were a failure (IIRC, some people were pretty impressed with the core NoSQL DB), but an exit to Apple may not have been all bad - although I would bet that Apple is not using the technology internally.

IME, my beef with the new NewSQL solutions is that they tend to be worse at both NoSQL and SQL workloads (specifically none are "the best" at OLAP and OLTP) except at one specific use case - global-replication. Getting global replication requires a cost (either monetary like Spanner, or performance wise like everything else), and global-replication just doesn't seem like a killer feature to me. Most times I need data in two datacenters globally, I wouldn't replicate the data anyways (don't want EU data in the US), and other times my SLAs just aren't tight enough to warrant the cost.

zzzcpan2• 9 years ago

I can't remember, was it FoundationDB that didn't have interactive client-server transactions too?

jasonwatkinspdx• 9 years ago

There was a much-hyped database a few years ago that failed to live up to the hype because of exactly that architecture. I can't even remember the name now when I was trying to find the post-mortem analysis for it in Google.

FoundationDB, acquired and taken internal by Apple. There were a few different blog posts about the layering aspect of it, with mixed opinions.

They built a transactionally consistent distributed database that was mysql client compatible and within 50% of mysql single node performance. If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that.

I'm also skeptical of a database claiming to be good at both OLAP and OLTP. One requires a column store, the other a row store. You can be half-decent as an OLAP store and good as a OLTP store.

This is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc.

I wouldn't call a single db that can do both OLAP and OLTP easy or a slam dunk, but I also wouldn't rule it out. VoltDB is well proven and worth looking at.

lobster_johnson• 9 years ago

If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that.

I think the parent is referring to performance. FoundationDB had a fast key-value store, but they supposedly weren't able to get good performance out of their SQL layer.

jsmthrowaway• 9 years ago

If creating that and getting it bought by Apple is failing, I hope I'm lucky enough to fail like that.

Careful, though, because one has to consider life after the acquisition if one loves the technology.

eloff4• 9 years ago

Thank you, it was FoundationDB I was thinking of, post-mortem analysis from the VoltDB guys here[1]. I'll contend they failed though because Apple bought them as an acqui-hire and killed the product. They were acquiring good engineers with a track record of executing well at a premium. It's a common recruiting strategy for tech companies. It doesn't mean that FoundationDB was a successful product - certainly it means it was not successful enough for the founders to risk turning down a lucrative acqui-hire deal.

his is a rigid and narrow view. There's lots of other design possibilities such as fractured mirrors, column decommission only within pages, etc.

Yes definitely I over-simplified to make my point, but I'll stand by it. OLAP and OLTP are different enough workloads and the engineering tradeoffs for being good at one of the other make it hard to excel at both. VoltDB is a row store, best suited for OLTP. Vertica is a column store best suited for OLAP. VoltDB can also make a decent OLAP database. Vertica does not make a good OLTP database. Both databases come out of Michael Stonebraker's work (although given the number of people required to make complex products like VoltDB and Vertica happen, it seems unfair to credit him unduly.)

But especially in the case of a sharded, distributed database where communication overhead is likely to dominate many types of queries (VoltDB, TiDB), one might not notice the inefficiency of using a poorly-suited design, like a row store for OLAP. This can make the differences less striking. This might also hide the inefficiency of basing everything on a generic KV store in TiDB.

[1]https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...

jandrewrogers• 9 years ago

There are other (widely used, in fact) alternatives to a classic column store or row store that exist for precisely the points you raise. It is possible to get something like (made up number) 80% of the OLTP performance of a row store and 80% of the OLAP performance of a column store if you employ some of the fancier page engine designs from the last decade, which is a good tradeoff for workload flexibility even if technically suboptimal for read and write exclusively.

However, I would argue that page layout doesn't matter that much in terms of system performance in very large scale-out systems. Other factors often dominate. I've seen row stores crush columnar stores for query performance, not because the row store was efficient (it wasn't) but because the inefficiency due to relative organization was offset by even greater efficiencies elsewhere in the architecture relative to the columnar store. Hence why using one of the "pretty good but not ideal for read and write" page engine designs and focusing elsewhere for query performance is a reasonable design choice.

I would agree that a generic KV-store is not a particularly efficient basis for a general-purpose database. You need to get much more query selectivity out of the content-addressability than a KV-store typically gives you for non-trivial data models.

manigandham• 9 years ago

CockroachDB is also built on a key/value layer using a Spanner type design. MemSQL is OLAP that does OLTP well, and Citus can make Postgres handle OLAP pretty well too (as used at Heap).

The article you're referring to is about Foundation DB written by John Hugg of VoltDB: https://www.voltdb.com/blog/2015/04/01/foundationdbs-lesson-...

The database world is rapidly changing and we're seeing new technologies that are enabling a lot more than we could do before. Just looking at some of the SQL on X engines alone shows massive improvements.

olegkikin• 9 years ago

Benchmark from half a year ago:

https://pingcap.github.io/blog/2017/05/23/perconalive17/#sys...

Previous discussions:

https://news.ycombinator.com/item?id=13298664

https://news.ycombinator.com/item?id=10180503

shenli35144• 9 years ago

We will release a new benchmark result soon.

itaifrenkel2• 9 years ago

How would you run spark on top of the production db without affecting its performance?

c4pt0r• 9 years ago

TiDB doesn't want to solve all the problems, no silver bullet, right? And I think it's all about workload, some analytical workload requires CPU/Network resource instead of I/O, and OLAP workload isn't that frequently. Storage layer and computing layer are separated in TiDB stack, I think it's possible for some workload.

itaifrenkel2• 9 years ago

The problem is that OLAP is run by analysts/business/data.science people and they may mess up with their queries/workloads. without isolation the production performance could impact the user expirience

galkk• 9 years ago

Their references to use cases from an article sounds funny "Migration from MySQL to TiDB to handle tens of millions of rows of data per day".

Come on, even Excel can handle tens of millions of rows of data per day.

buryat• 9 years ago

Excel can not handle more than 1048576 rows

https://support.office.com/en-us/article/Excel-specification...

aneutron• 9 years ago

Not to mention that file locking would make it literally impossible to concurrently edit the file.

otterley• 9 years ago

Contrary to the assertion in the press release, there's no evidence Abraham Lincoln ever said, "the best way to predict the future is to create it." That would be Alan Kay.

https://quoteinvestigator.com/2012/09/27/invent-the-future/

jinqueeny• 9 years ago

That’s interesting! I googled it too, but found out it could be Abraham Lincoln or Peter Drucker...

qq66• 9 years ago

Anyone who has read any of Lincoln's papers knows that simply isn't the way he would talk, or really anyone from his time period. The language that we use today around innovation, envisioning futures, boldly creating the world we want to live in, etc. is post-WWII.

scroot• 9 years ago

This is totally bizarre. It must be a joke, right?

TiDB – a global scale distributed DB

Related Stories