sougou

Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.

This is part 7: Discovery and Propagation. With this, we complete everything about the protocol part of Generalized Consensus. There are a few more advanced topics to cover: They are needed for running in production.

One of our goals is to make this approach work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.

Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.

This is part 6: It covers Revocation and Candidacy, the prerequisites for a leader change. We explain how to revoke previous leaderships and recruit for a new candidacy. It's a bit complex, but there are plenty of animations to help you along.

One of our goals is to make this approach work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.

Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.

This is part 5: This one covers how to safely handle multiple coordinators racing to take action. We explore term numbers, coordinators as separate agents, and lock-free approaches.

One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.

Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.

This is part 4: This one has a nice animation that shows how requests get fulfilled.

One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.

I'll post here as I release the subsequent parts.

Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.

This is part 3: Governing Rules. This is the foundation for the rest of the series. We'll be repetitively applying these rules to map out all parts of the protocol. Also, every system that I know of implicitly follows these rules.

One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.

I'll post here as I release the subsequent parts.

Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems. I've released two parts so far. They explain the problem and the requirements. You can follow the link to the second part from the first part.

One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.

I'll post here as I release the subsequent parts.

I oversaw this work, and I'm open to feedback on how things can be improved. There are some factors that make this particular situation different:

This was an LLM assisted translation of the C parser from Postgres, not something from the ground up.

For work of this magnitude, you cannot review line by line. The only thing we could do was to establish a process to ensure correctness.

We did control the process carefully. It was a daily toil. This is why it took two months.

We've ported most of the tests from Postgres. Enough to be confident that it works correctly.

Also, we are in the early stages for Multigres. We intend to do more bulk copies and bulk translations like this from other projects, especially Vitess. We'll incorporate any possible improvements here.

The author is working on a blog post explaining the entire process and its pitfalls. Please be on the lookout.

I was personally amazed at how much we could achieve using LLM. Of course, this wouldn't have been possible without a certain level of skill. This person exceeds all expectations listed here: https://github.com/multigres/multigres/discussions/78.

We absolutely need this!!! I vote yes :).

You won't believe what a nightmare it was to work with transactionless DDLs in MySQL. Transactional DDL will be a dream come true for Vitess: we can throw away all the hacks we had to do for MySQL's sake.

I also see such a clean 2PC API. This was another huge mess on the MySQL side.

The Vitess architecture was traditionally built in a database agnostic fashion. Most of its features should port smoothly over to Postgres.

The cool features that I can think of: A formal sharding scheme based on relational foundations, a fairly advanced query analyzer and routing engine capable of cross-shard functionality, HA and durability, abilities to reshard safely, seamless migrations, etc.

It's a pretty big list of capabilities.

Of course, but not right away. We'd like to get the project to a decent stable state first. Then, we'll open it up.

PgDog looks very similar to Vitess in terms of approach.

It will be a proxy layer. Since Vitess has proven itself with this approach, I don't see a reason to try anything drastically different. As for Planetscale, I've posted my thoughts on X and LI :).

Hello HN, co-creator of Vitess here. The goal of Multigres is to build an open source adaptation of Vitess for Postgres.

In terms of project plan, we're still deciding. One option is to fork and retrofit. This will give us immedate results, but we'll be inherting quite a bit of legacy that we'll have to overcome later. The other option is to start with a clean slate and move pieces from the original Vitess code. This will ensure that Multigres has the Postgres DNA. We'll also avoid inheriting legacy features. Let us know your thoughts on this.

I'll try my best to answer any other questions you may have.

It's not just failure detection. A write to EBS is at least two additonal network hops. The first one is to get to the machine for the initial write, and the second is for that write to be propagated to another machine for durability. Multiply this by the number of IOPS required to complete a database transaction.

The way to reason about this is: If a customer had a huge business, what key would they shard by? Then choose that as the sharding key for all customers.

This is still in our plans. We're likely to launch by end of year.

We've come a long way in Vitess in terms of usability. And we're adding even more tooling to support onboarding and migrations.

Github is evaluating Vitess. They'll probably talk about it if and when they deploy it into production.

Vitess has applied to join the CNCF. I recently made a presentation to the Storage Working Group: https://docs.google.com/presentation/d/1xgDO8zr3Tmic4NV9DOp_...

Right. Vitess was built by YouTube to solve its own scalability problems. It provides sharding and cluster-management functionality. In a way, it's orthogonal. We've talked about the possibility of making Vitess work on top of CloudSQL. Maybe it will happen some day.

You also have the option of using local storage and combine it with a consensus protocol to keep the data distributed. You can actually achieve better durability than mainframes.

Spanner uses cross-datacenter Paxos. Your data won't be lost even if an entire datacenter goes dark.

For Vitess (http://vitess.io), we use semi-sync replication that always ensures that at least one other machine has the data.

Hi, thanks for the compliments :).

MySQL group replication has some issues: It can fail your commits if multiple masters have conflicting transactions, which is a problem for cross-shard distributed transactions. Additionally, group replication is too chatty and doesn't work well cross-dc.

I actually have a counter-proposal that addresses the above concerns here: http://ssougou.blogspot.com/2016/09/distributed-durability-i....

Disclaimer: I work on Vitess.

I'm actually a fan of TiDB and Cockroach. I've met engineers from both, and they're super-sharp.

I am biased towards something like Vitess that builds on top of existing tech. The main advantage is that we can push-down the work to the lowest level, and leverage efficiencies that are already built in MySQL.

But the newer architectures offer better consistency models. Some customers may care more about that.

In the long run, I think these trade-offs should converge. Overall, any NewSQL is a better alternative than using traditional key-value stores, because it gives you better functionality while not giving up on scalability.

I'm the author of the blog. Thanks for the posts and interest shown.

This proposal is a generalization of the core Paxos algorithm. It's therefore orthogonal to all customizations and adaptations. I believe that they can all benefit from it, including RAFT.

If you read the thread on raft-dev, you'll see that Heidi Howard has independently come up with the same idea, and is working on a proof for it.