The Great Re-shard: adding Postgres capacity (again) with zero downtime (2023)

https://www.notion.com/blog/the-great-re-shard

by unchar1 • 1 year ago

30 8 1 year ago

winrid2• 1 year ago

Wonder what hardware they are using and what the actual iops numbers are

tucnak2• 1 year ago

I wouldn't be surprised if it's some stock AWS instance type, probably not even NVMe-enabled, or some kind of metal instance bottlenecked by EBS. The downside of Cloud push in SaaS is that they were unable to tap into the ever-changing storage landscape, NVMe-oF, and so on. This is mostly gun-in-foot matter more than anything.

jitl2• 1 year ago

I work at Notion.

It’s stock RDS on EBS, which as you say is super slow compared to Genuine Local Storage. Any large scan or large result set query that doesn’t fall into the Postgres cache or the underlying EBS cache is frustratingly slow.

However almost all of our reads are “get row by ID” lookups that hit memcached. Our query pattern is mostly recursive graph traversal which Postgres is bad at, with our table size even a 20x faster disk wouldn’t make it feasible to do in SQL. We also copy paste “notion database” data in to a stateful caching service, but that also needs specialized indexing for user defined schema, another thing Postgres can’t handle (my last project was an improvement to that service).

The place we really suffer from the EBS slowness is with full table scan data migrations, stuff like “for all blocks in Notion, block = f(block)”. For this we use “Big Data” approach reading from DB dumps in S3 instead of Postgres directly.

Ultimately I’d prefer to serve every read from your device’s local storage, which is my current project.

The Great Re-shard: adding Postgres capacity (again) with zero downtime (2023)

Related Stories