jinqueeny

I am so sorry about it. The website was down due to some server issues after launch. The problem has is fixed. Hopefully, it can work as you expected.

Sorry. I will restrain myself.

Sorry if the post sounds too marketing-heavy. I'm sharing this case study because it addresses a common technical challenge: scaling a high-traffic application (Kwai, a video platform) from traditional MySQL to a distributed database system. It describes the journey from managing 300+ MySQL shards to consolidating into a single 400TB cluster using OceanBase, handling over 1 million QPS during peak loads. While the post is oriented towards promoting OceanBase, it resonated with developers because it provides specific technical details about real-world scaling problems, architecture decisions, and performance metrics that many engineering teams face when growing beyond traditional MySQL setups.

It has a lot of use cases in the mining industry, such as monitoring mine water, ventilation systems, fluids in tanks and pipes, distance and angle measurements, remote operation of actuators and equipment, environmental monitoring, etc.

ops, sorry. Here it is: https://github.com/myscale/ChatData

In short, Apache Pulsar beats Kafka:

- 2.5x Maximum Throughput Compared to Kafka

- 100x Lower Single-digit Publish Latency than Kafka

- 1.5x Faster Historical Read Rate than Kafka

Disclaimer: PingCAPer here. Thanks for your attention to PingCAP and TiDB. It’s true that we are on HN front page for many times - that’s incredible recognition for the value of our technical content, and we are humbled and thankful. We take great pride in our work, and have always committed to valuable, accurate and insightful content.

As for referring our PVLDB as “the first industry paper”, thanks @karsinkk for pointing out. We admit that the original description was inaccurate and just updated it to be “This is the first paper in the industry to describe the Raft-based implementation of a distributed Hybrid Transactional/Analytical Processing (HTAP) database.”

However, we have never claimed ourselves as “the best, the fastest.” If you find such occurrences and think they are inaccurate, please do share with us and we will correct the same ASAP. As pointed in https://pingcap.com/blog/9-why's-to-ask-when-evaluating-a-di..., “There's no single technology that can be the elixir to all your problems. The database realm is no different. If your data can fit on a single MySQL instance without too much pressure on your server, or if your performance requirement for complex queries isn't high, then a distributed database may not be a good choice. Choosing to use a distributed database typically means additional maintenance cost, which may not be worthwhile for small workloads. “

Regarding your comparison between TiDB and MySQL, there are differences between MySQL and TiDB as stated here: https://opensource.com/article/18/11/key-differences-between.... If you are still interested in trying out TiDB with your real life workload, we are very happy to help you. Please reach out to us in our community Slack: https://slack.tidb.io/invite?team=tidb-community&channel=eve...

TiDB has a similar case study: Queries over 1.3 Trillion Rows of Data Within Milliseconds of Response Time at Zhihu.com https://pingcap.com/success-stories/lesson-learned-from-quer...

The latest stats in the same case scenario (already-read posts) Zhihu is:

- 2.6 Trillion Rows

- 560TB data

- 200 TiKV instances

The link to the paper (on VLDB 2020) is now available: http://www.vldb.org/pvldb/vol13/p3072-huang.pdf

I am wondering have you considered the open source NewSQL database solutions like TiDB, CockroachDB? They have the best of both traditional RDBMS and NoSQL and would be the perfect choice to the hyper-growth scenarios.

TiDB could be considered as a scale-out MySQL and CRDB a scale-out Postgres. They are all Spanner-inspired solutions that can help avoid manual sharding.

Thanks! Posting the case study here for more information: https://pingcap.com/success-stories/lesson-learned-from-quer...

For the impatient: “The etcd key-value store is a distributed database based on the Raft consensus algorithm. In our 2014 analysis, we found that etcd 0.4.1 exhibited stale reads by default. We returned to etcd, now at version 3.4.3, to investigate its safety properties in detail. We found that key-value operations appear to be strict serializable, and that watches deliver every change to a key in order. However, etcd locks are fundamentally unsafe, and those risks were exacerbated by a bug which failed to check lease validity after waiting for a lock.”

This case study has been updated: https://pingcap.com/success-stories/why-we-chose-tidb-over-o...

Thanks for the comment! CockroachDB is now under BSL and CCL licenses. Different users have different opinions but you are right, we need to be accurate about it. Please give me some time to consult our user and see whether we can update this case study.

You can check out the case studies from our users here: https://pingcap.com/success-stories/

Want to make your own Golang apps run in a browser? Here’s what we learned ...

We had a lot of fun—and learned a lot—from our little adventure at TiDB Hackathon 2019. If you want to make your own Golang applications run in a browser, here are some suggestions:

As [WebAssembly Weekly - Issue #100](https://wasmweekly.news/issue-100/) mentioned, "WebAssembly support for Go applications is very much in its infancy.” For example, Golang hasn't fully supported WASI, and goleveldb doesn’t support Wasm/js. Be discreet before you go too deep, and clearly understand the use cases for your application.

Make sure your application doesn't have third-party platform-specific dependencies that can’t be compiled to Wasm.

Browsers don’t allow port listening and file operations, so you’ll have to work around those limitations.

This is a demo site to show how you can use TiDB-Wasm to run TiDB SQL statements your browser.

https://github.com/pingcap/tidb/pull/13069

Known Limitations:

- TiDB-Wasm includes TiDB server only. TiDB is the stateless MySQL component of the TiDB platform (check other components in the TiDB Architecture). Compiled to Wasm, this tool is only suitable for experimenting with SQL and testing out TiDB compatibility with MySQL queries.

- The Wasm binary file is close to 70 MB and compressed to 13 MB. The database and the data you create in TiDB-Wasm are written to memory, and therefore consume a lot of memory resources.

- TiDB-Wasm does not support most of the web browsers on mobile devices yet.

- The tool currently cannot demonstrate multi-node TiDB deployments. This means you can’t play with TiDB’s titanium-solid resiliency, test doing rolling updates, or online scale-out and scale-in.

Thanks to Wasm (https://webassembly.org/) for making it possible to build desktop-quality applications in the web browser.

Thanks to the Golang for supporting Wasm, and making it possible to run performant Golang applications or even databases in the web browser.

Thanks to the Ti-Cool team in the TiDB community for contributing TiDB-Wasm in the 2019 TiDB Hackathon, making running TiDB in local model with by embedding GoLevelDB in the web browser possible. The team members are currently working on a technical blog to share how they made it. Stay tuned!

tl;dr: Made possible by introducing a non-voting role (Raft learner) in the Raft group.

FYI: this is just a PoC and is very early in the stage :)

thanks

It's built on top of Tantivy (https://github.com/tantivy-search/tantivy) that implements Tha Raft Consensus Algorithm (https://raft.github.io/) by raft-rs (https://github.com/tikv/grpc-rs) and The gRPC (HTTP/2 + Protocol Buffers) by grpc-rs (https://github.com/tikv/grpc-rs) and rust-protobuf (https://github.com/stepancheg/rust-protobuf).

Thanks for sharing and I am sorry for your project. It was a cool idea. Any idea why no actual adoption? Is this possibly because of the product-market fit mentioned in this post?

I am thinking aloud here: maybe the product-market fit, project-community fit and the value-market fit were not so clearly separated or sequential but more interrelated from the very beginning. You have to find the value-market fit and define the business problem first and then the community will thrive? The value to the community is not only what the project can bring to the business, but also the value this project can bring to its contributors and adopters.

Yes, the number of contributing companies to the project is a good metric too. One more thing is their depth and significance of the contribution, though it might be very difficult to measure.

Very informative. Curious of what others think of the measurements of the three pillars of open source:

- Project-Community Fit: GitHub stars

- Product-Market Fit: Downloads

- Value Market Fit: Revenue

As a matter of fact, we recently released the 1.0 GA version of TiDB Operator: https://github.com/pingcap/tidb-operator

0.3.0 gfx-hal API is recently released: https://www.reddit.com/r/rust/comments/cnvfrm/gfxhal03_relea...

Shameless self-plug: If you are looking for an open-source scale-out MySQL that supports both multiple writes and multiple reads, feel free to give TiDB (https://github.com/pingcap/tidb) a try.

TiKV is built on top of RocksDB, see our chief engineer’s speech of how we used RocksDB in TiKV: https://pingcap.com/blog/2017-09-15-rocksdbintikv/

Because of the write amplification issue of RocksDB, we build a RocksDB plugin, TiTan. Here is its design and implementation detail: https://pingcap.com/blog/titan-storage-engine-design-and-imp...

Impressively informative! Regarding the install/deployment about TiDB, we have build tools such as open-source tools such as TiDB-Ansible for on-prem and TiDB-Operator for kubernetes. Here is a list of deployment instructions:

Ansible: https://pingcap.com/docs/dev/how-to/deploy/orchestrated/ansi...

Kubernetes (using TiDB-Operator): https://pingcap.com/docs/dev/how-to/deploy/orchestrated/kube...

Terraform on AWS: https://github.com/pingcap/tidb-operator/tree/master/deploy/...

Docker: https://pingcap.com/docs/dev/how-to/deploy/orchestrated/dock...

Binary: https://pingcap.com/docs/dev/how-to/deploy/from-tarball/prod...