Neki – Sharded Postgres by the team behind Vitess

https://planetscale.com/blog/announcing-neki

by thdxr • 12 months ago

251 42 12 months ago

shoeb00m• 12 months ago

Looks like there is two ongoing vitess for postgres projects. Hopefully this competition leads to a better postgres ecosystem.

https://supabase.com/blog/multigres-vitess-for-postgres

qaq2• 12 months ago

Supabase also working on OrioleDB

eatonphil3• 12 months ago

OrioleDB is not about sharding, it's about the storage layer.

n2d4• 12 months ago

There is also pgdog by the author of pgcat: https://pgdog.dev

dangoodmanUT• 12 months ago

It gets more spicy when you realize the founder of vitess, also the founder of planet scale, left planet scale to build this at supabase

samlambert• 12 months ago

he left PlanetScale 4 years ago.

hubertzhang• 12 months ago

Does Neki still need sharding key in query, just like Citus?

rockwotj• 12 months ago

If it’s like vitess then no, but IIRC you get relaxed consistency across shards

n_u• 11 months ago

Regarding cross-shard consistency, there's a doc here[1] on the options. In short you can just send the writes to all replicas, you can disallow cross-replica transactions (read and write), use two-phase commit to perform a distributed transaction or use their own hybrid approach that they set as the default.

[1] https://vitess.io/docs/20.0/user-guides/configuration-advanc...

ozgrakkurt• 12 months ago

Is anyone working on replacing postgres?

Feels like it might be very useful since a lot of new technologies came out since spinning disks.

erpellan2• 12 months ago

If you look at the changes that have been made to Postgres, and continue to be made, the answer is yes.

The Postgres team is working on replacing Postgres. With even better Postgres.

bddicken2• 12 months ago

The Postgres team incorporating io_uring into PG 18 is a good example of this: https://pganalyze.com/blog/postgres-18-async-io.

zozbot234• 11 months ago

inb4 "It's the best Postgres we've ever made!"

eatonphil3• 12 months ago

Who isn't? Cockroach rewrote Postgres in Go. CedarDB rewrote Postgres in C++.

And then to lesser degrees you've got Yugabyte, AlloyDB, and Aurora DSQL (and certainly more I'm forgetting) that only replace parts of Postgres.

vladich• 11 months ago

Both Cockroach and CedarDB didn't rewrite anything, they built stuff from scratch. Just used the same client protocol. There are a bunch of other unrelated databases using Postgres protocol btw.

gethly• 11 months ago

Feels like it might be very useful since a lot of new technologies came out since spinning disks.

The MVCC that Postgres uses(and no one else) is like 50yo outdated concept they still cling to. So just by virtue of that, it makes PGSQL the most archaic db on the market nowadays.

I never understood why PGSQL had so many fanboys, yet every major tech company always ditches it for mysql... i guess it is the case of "Any man who must say, "I am the King", is no true king." type of thing. People have to make a lot of noise to excuse their bad choices so they don't have to admit making a mistake.

erpellan2• 11 months ago

I can’t believe they still make processors out of sand. Talk about outdated technology.

Sarcasm aside, a great many projects started on MySQL and moved to postgres. As did projects using mongo, couchdb, firebase, oracle etc etc…

And I’m sure many projects switched away from Postgres to other technologies. Right tool for the job at hand.

lmz2• 11 months ago

By MVCC you mean the kind of MVCC that keeps old versions in the same space / requires vacuum? Because I'm pretty sure Oracle also does multiversioning using their undo log / rollback segment.

franckpachot• 11 months ago

The companies that attempt to replace PostgreSQL do so not to replace PostgreSQL itself, but to replace Oracle.

atombender2• 12 months ago

This is exciting. The announcement says it will be open source. I really hope that this includes a functionally complete control plane so you realistically self-host.

I looked Neon recently, and it appears that it's designed as a SaaS product from the outset; while it is technically possible to self-host the individual components of the architecture, it does not look trivial, in large part because the control plane is closed source (and probably extremely specific to Neon's SaaS operations).

benjiro2• 12 months ago

Your probably better off with the original flavor (the guy that made Vitesse)

https://multigres.com/

He is making a open source version of porting Vitesse to Postgres.

bddicken2• 11 months ago

Why? Neki is built by the the engineers who have built, maintain, and operate massive-scale Vitess databases.

fnord123• 11 months ago

Discussed on The Changelog podcast: https://changelog.com/podcast/651

kyrra• 12 months ago

They did a interview 2 weeks ago about this on the changelog podcast with Sugu Sougoumarane, the co-creator of Vitess, who is at Supabase now. Watching the competition year will be interesting

https://youtu.be/y1aq8RsnJeI

josevalerio• 12 months ago

Just FYI that interview is for Multigres, a sharded Postgres in development by Supabase.

Sugu hasn’t been at Planetscale for a few years

https://github.com/multigres/multigres

purpleidea• 12 months ago

Maybe make the announcement after you've actually released code?

Today I'm announcing I've cured cancer. Well not yet, but coming soon hopefully!

tpetry• 11 months ago

More like: We‘re announcing to work on a cure in cancer! Isn‘t that aweslme? No, we dont have anything yet. But we started working on it. ETA? When its done. We dont share roadmaps with outsiders.

crowcroft• 11 months ago

Probably want to try front run comms with Supabase getting into this as well.

Reubend• 11 months ago

Very cool, and I'm looking forward to using this. But unfortunately it seems like it's not actually released yet?

I'm a little surprised to hear that PlanetScale is doing the work to make this considering I thought their entire system was based on Vitess. Maybe the demand for Postgres compatible DBs is so high nowadays that they need to offer compatibility for customers that don't want to port their apps to MySQL's syntax?

abrkn• 11 months ago

I love this website!

saurik2• 11 months ago

I wonder how this compares to YugabyteDB?

v5v3• 11 months ago

Yugabyte is a 'postgresql compatible' database.

These are forks or extensions of Postgresql.

saurik2• 11 months ago

YugabyteDB is also a fork of PostgreSQL. Last year, they got around to rebasing from 11 to 15, and spent some time working on better isolating their changes to make that easier going forward.

https://www.yugabyte.com/blog/yugabytedb-moves-beyond-postgr...

When we set out to make YugabyteDB Postgres-compatible, we took a fork of Postgres, and modified all of the operations that use shared memory or storage to instead talk to our LSM- and Raft-based distributed storage and transaction layer.

Neki – Sharded Postgres by the team behind Vitess

Related Stories