PolarDB, yet another open source database system based on PostgreSQL

https://github.com/alibaba/PolarDB-for-PostgreSQL

by jinqueeny • 5 years ago

143 85 5 years ago

github.com

qaq3• 5 years ago

This field is getting crowded wonder if there will be some type of consolidation.

sail0rm00n• 5 years ago

There’s no need for consolidation when Postgres already exists.

speedgoose4• 5 years ago

Sometimes I dream about a highly available multi primary postgresql.

qaq3• 5 years ago

well some use cases need something that is distributed like PolarDB, YugobyteDB, CockroachDB etc.

tzumby• 5 years ago

“high availability through Paxos based replication”. I thought Paxos is supposed to be a CP system.

jlokier• 5 years ago

You have N >= 3 nodes, or N >= 2 and a non-compute arbitrator. One of them goes down, stops responding. You still have quorum, data processes continue just fine. That's high-availability in a CP system.

chousuke2• 5 years ago

HA isn't really about 100% availability. Any single system that promises such is extremely likely to be misleading you somehow. Your in-flight query is going to get interrupted no matter how fancy your clustering is, and I struggle to even come up with hypothetical use cases where this is something you can't afford to have happen, ever.

All you need is that in the event of a failure the clustered system can still recover quickly enough (to a well-defined state!) that the application layer can deal with the transient failure without significant impact on users, maintaining the illusion of availability.

chii• 5 years ago

I struggle to even come up with hypothetical use cases where this is something you can't afford to have happen, ever.

a rocket is using this query to adjust their trusters ;)

mahkeiro• 5 years ago

It seems that Alibaba is going more open source than Amazon for their clouds services. That’s interesting and they may use this approach as a key differentiator.

unityByFreedom3• 5 years ago

Based on what metric? Amazon took over Elastic's open source server and another major one IIRC. That was huge. PostgreSQL is not about to disappear.

greatpatton• 5 years ago

They did it only after ES decided to abandon the Apache license. I doubt that Amazon would have done it if they were not forced. Redshift is also Postgres based and Amazon never released something. So based on that metric Alibaba is more open.

wanghq2• 5 years ago

Oh, what's going on here?

OceanBase, the database of Alibaba's fintech company Ant Group, will be open-source soon, possibly as early as June 1, according to Sina Tech.

https://cntechpost.com/2021/05/27/ant-groups-in-house-databa...

Also the team published many papers about PolarDB.

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C48&q=%22...

wejick2• 5 years ago

Looks like they're different technologies, found that oceandb has been offered in AliCloud under ApsaraDB umbrella https://www.alibabacloud.com/product/oceanbase

foobarbazetc• 5 years ago

Alibaba is huge and different teams have different goals.

That one is for fintech.

yftsui• 5 years ago

These are just KPI projects as their performance are judged by meaningless metrics. OceanBase WAS open sourced, after the news reports and publicity, Alibaba deleted all code and replaced it with a Chinese announcement says it will no longer be an open source project.

wanghq2• 5 years ago

Sorry I didn't make myself clear. I was surprised because two databases from Alibaba are open sourced or to be open sourced.

ioltas• 5 years ago

Looking at their code tree, this is based on 11.2 (https://www.postgresql.org/docs/11/release-11-2.html): https://github.com/alibaba/PolarDB-for-PostgreSQL/blob/maste...

So this is code from two years ago missing stability fixes from upstream up to 11.12, at short sight.

the-dude• 5 years ago

Are there any stability fixes you have in mind? In my recollection, there is not much instability in PG releases.

pella2• 5 years ago

IMHO: with the latest fix -> more stable

search for "crash":

https://www.postgresql.org/docs/release/11.3/ ( 11x )

https://www.postgresql.org/docs/release/11.4/ ( 4x )

https://www.postgresql.org/docs/release/11.5/ ( 1x )

https://www.postgresql.org/docs/release/11.6/ ( 3x )

https://www.postgresql.org/docs/release/11.7/ ( 14x )

https://www.postgresql.org/docs/release/11.8/ ( 5x )

https://www.postgresql.org/docs/release/11.9/ ( 6x )

https://www.postgresql.org/docs/release/11.10/ ( 5x )

https://www.postgresql.org/docs/release/11.11/ ( 3x )

https://www.postgresql.org/docs/release/11.12/ ( 5x )

mb4nck3• 5 years ago

Looks like it, yeah.

Also, Postgres 12 introduced pluggable storage, which might help to implement a shared-nothing architecture without huge changes to vanilla Postgres (I haven't looked at how large their delta is)

jeff-davis2• 5 years ago

Citus Data enables scale out, while also being a pure extension. That means you can upgrade Postgres like normal to the latest point release using whatever normal upgrade process you want (e.g. OS packages).

It has worked for a long time without the need for Postgres 12. However, the new APIs introduced in v12 did enable us to offer columnar compression as an option, which complements a lot of scale-out use cases.

See: https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-...

I believe using the extension facilities of Postgres is far superior to a fork in the medium to long term.

(Disclaimer: I work for Citus, and on columnar compression.)

he0001• 5 years ago

What more db’s are based on Postgres?

jeltz• 5 years ago

Greenplum, Postgres-XL and PipelineDB.

ziftface• 5 years ago

I'm still really upset that confluence bought and ruined pipeline db. It solved my problem perfectly and I haven't seen one yet that fits what I needed as well.

ggordan• 5 years ago

I think "based on" is more like implemented on top of, similar to timescaledb

helltone• 5 years ago

Yellowbrick

mkl95• 5 years ago

TimescaleDB.

hashhar• 5 years ago

Greenplum

Hadapt

Netezza

PipelineDb

Postgres-XL

Redshift

AgensGraph

TimescaleDB

Fujitsu Enterprise Postgres

PolarDB

CrunchyBridge

craigkerstiens2• 5 years ago

A quick note on Crunchy Bridge, it is pure Postgres, not "based" on Postgres. We haven't forked or modified code at all. And some of the above (Timescale) are extensions which do not change Postgres, but hook into the extension APIs.

rad_gruchalski7• 5 years ago

Yugabyte

zzzeek• 5 years ago

EdgeDB

your_challenger• 5 years ago

Looks like EdgeDB hasn't done enough marketing. No one has mentioned EdgeDB except you. Well we're (I love EdgeDB) still in beta.

darrenbkl• 5 years ago

Cockroachdb

welder• 5 years ago

No, this one's built from scratch in Go and based off LevelDB/RocksDB. They support Postgres wire protocol, but not based off any Postgres code.

devit• 5 years ago

Can anybody comment on whether this actually works properly and does what it claims to do?

killingtime74• 5 years ago

Why wouldn’t I use cockroachDB?

mb4nck3• 5 years ago

I don't know, maybe you should.

CockroachDB seems to be a distributed database system written in Go which has implemented a Postgres query/wire protocol compatibility layer.

PolarDB is a Postgres fork actually using the Postgres codebase and extending it to a distributed database system. Maybe one day they can unfork because it's possible to implement PolarDB on top of Postgres as an extension and/or they contribute/get all their changes into Postgres core.

rad_gruchalski7• 5 years ago

Maybe because you need the as-a-service aspect and the licensing does not allow it.

joelp• 5 years ago

What are people/companies using currently to make their postgres database distributed these days?

Currently running my app + db on Heroku in the EU and would like to scale out since latency is abysmal for users in Australia and Asia.

arthurcolle2• 5 years ago

I think Citus is one option. Or possibly TimescaleDB

manigandham2• 5 years ago

Those are meant for a single location, not cross-region distribution.

wirelesspotat• 5 years ago

fly.io[1] supports Postgres clusters with read replicas in different cities[2]

It looks like AWS's Aurora Postgres doesn't support cross-region read replicas, but apparently their Aurora "Global Database" offering does[3]

- [1] https://fly.io

- [2] https://fly.io/docs/reference/postgres/#scaling-horizontally...

- [3] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

craigkerstiens2• 5 years ago

For fully managed (similar to Heroku Postgres) Crunchy Bridge [1] supports replicas across regions including say EU to AU. Which would actually couple really well with Fly.io for the app side.

- [1] https://www.crunchydata.com/products/crunchy-bridge/

manigandham2• 5 years ago

The simple solution is to run Postgres replicas in those regions and do local reads for your app while directing the writes to the master region. This works well for read-heavy apps, and you can also put the master in a geographically central location to help with write latency.

If you need cross-region multi-master then I recommend something like CockroachDB or Yugabyte that have regional distribution as a core feature.

arcturus17• 5 years ago

I’m glancing over the Heroku Postgres docs and they seem to offer easy to set up read replicas called “followers”. Wouldn’t this work for you?

iampims• 5 years ago

Heroku is only available in EU and US east. Latency from Australia is somewhat unavoidable. You can maybe reduce it by using Cloudflare or similar to terminate TLS as close to your users as possible.

raarts• 5 years ago

What are the differences between this and YugobyteDB?

wejick2• 5 years ago

Yugabyte is different DB technology than postgres but with PG compatibility layer, means you can use existing postgres query/application, the rest is different beast. This one is more comparable with citus, an PG extension. Meaning it runs on top postgres. However from their github page they also offer patched version of PG which I suppose offer some tighter integration with the extension.

JoelJacobson3• 5 years ago

No commit history? It would have been interesting to follow the development work to see what changes they've made compared to PostgreSQL.

llaolleh• 5 years ago

They are following the Gaussian approach. Show only the final result!

antoncohen• 5 years ago

This is effectively required when open sourcing something that was previously internal. I've done it at a company. There could be company internal things that leak in old revisions of code and commit messages. Even if the latest commit on master is clean, it is really hard to know that every revisions, throughout the whole history is clean.

JoelJacobson3• 5 years ago

Good point. In such a case, I think it would still be nice if at least trying to split the total change into a few separate commits, that builds upon each other. That way the commit log could be a good start to look at for someone who wants to understand the code base.

rad_gruchalski7• 5 years ago

I'd love to try this out. What I'm missing though is at least some Docker based deployment not involving building stuff from sources and reinventing distributed architecture myself.

PolarDB, yet another open source database system based on PostgreSQL

Related Stories