Building and scaling Notion's data lake

https://www.notion.so/de-de/blog/building-and-scaling-notions-data-lake

by alexzeitler • 2 years ago

239 79 2 years ago

Iceberg and Delta Lake, on the other hand, weren’t optimized for our update-heavy workload when we considered them in 2022

"when we considered them in 2022" is significant here because both Iceberg and Delta Lake have made rapid progress since then. I talk to a lot of companies making this decision and the consensus is swinging towards Iceberg. If they're already heavy Databricks users, then Delta is the obvious choice.

For anyone that missed it, Databricks acquired Tabular[0] (which was founded by the creators of Iceberg). The public facing story is that both projects will continue independently and I really hope that's true.

Shameless plug: this is the same infrastructure we're using at Definite[1] and we're betting a lot of companies want a setup like this, but can't afford to build it themselves. It's radically cheaper then the standard Snowflake + Fivetran + Looker stack and works day one. A lot of companies just want dashboards and it's pretty ridiculous the hoops you need to jump thru to get them running.

We use iceberg for storage, duckdb as a query engine, a few open source projects for ETL and built a frontend to manage it all and create dashboards.

0 - https://www.definite.app/blog/databricks-tabular-acquisition

1 - https://www.youtube.com/watch?v=7FAJLc3k2Fo

whinvik• 2 years ago

Is there any advantage to having both a Data Lake setup as well as Snowflake. Why would one also want Snowflake after doing such an extensive data lake setup?

luizfwolf• 2 years ago

Saving money 100% also lower latency on distributed access. Accessing file partitioned S3 doesn’t require to spin a warehouse and wait for your query to go on a queue, so if every job runs in like k8s you don’t have to manage resources and auto scale in snowflake is a “paid feature”

I believe just not having to handle a query queue system is already.

mritchie7126• 2 years ago

Many BI / analytics tools don't have great support for Data Lakes, so part of the reason could be supporting those tools (e.g. they still load some of their data to snowflake to power BI / dashboards)

Lucasoato2• 2 years ago

We've solved that issue with Trino. Superset and a lot of other BI tools support connection to it and it's a very cost efficient engine (compared to DWH solutions). Another way to go even cheaper is using Athena, if you're on AWS.

kdazzle• 2 years ago

For one, Snowflake is expensive (you pay for the convenience and simplicity) and the data in there is usually stored in S3 buckets that Snowflake owns (and they dont pass along any discounts that they get from AWS for the cost of that storage).

adolph• 2 years ago

They seem to be doing lots of work but I don’t understand what customer value this creates.

What does a backing data lake afford a Notion user that can’t be done in a similar product, like Obsidian?

jpalomaki• 2 years ago

From the article: "Unlock AI, Search, and other product use cases that require denormalized data"

SOLAR_FIELDS2• 2 years ago

Beyond the features that the sibling comment mentioned, this kind of data isn’t really for end users. It’s a way that you can package it up, “anonymize” it, and sell the data to interested parties.

sghiassy• 2 years ago

Is that why they’re putting images in Postgres? I don’t understand that design decision yet.

Ozzie_osman• 2 years ago

For someone like Notion, they probably aren't selling this data. The primary use case is internally for analysis (eg product usage, business analysis, etc).

It can also be used to train AI models, of course.

ambicapter• 2 years ago

1st paragraph: "Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake."

bastawhiz6• 2 years ago

The whole point of a data warehouse is that you can rapidly query a huge amount of data with ad hoc queries.

When your data is in Postgres, running an arbitrary query might take hours or days (or longer). Postgres does very poorly for queries that read huge amounts of data when there's no preexisting index (and you're not going to be building one-off indexes for ad hoc queries—that defeats the point). A data warehouse is slower for basic queries but substantially faster for queries that run against terabytes or petabytes of data.

I can imagine some use cases at Notion:

- You want to know the most popular syntax highlighting languages

- You're searching for data corruption, where blocks form a cycle

- You're looking for users who are committing fraud or abuse (like using bots in violation of your tos)

SOLAR_FIELDS2• 2 years ago

They didn’t say the quiet part out loud, which is almost certainly that the Fivetran and Snowflake bills for what they were doing were probably enormous and those were undoubtedly what got management’s attention about fixing this.

rorymalcolm• 2 years ago

Found this comment (from Fivetran's CEO, so, with that in mind) regarding this article enlightening regarding the costs they were facing here https://twitter.com/frasergeorgew/status/1808326803796512865

pbd• 2 years ago

Snowflake as destination is very very easy to work with on fivetran. Fivetran didn't have S3 as destination till late 2022. So it literally forces you to use one of BQ, Snowflake, redshift as destination. So fivetran CEO's defence is pretty stupid.

sneak• 2 years ago

I thought the quiet part was that they are data mining their customer data (and disclosing it to multiple third parties) because it’s not E2EE and they can read everyone’s private and proprietary notes.

Otherwise, this is the perfect app for sharding/horizontal scalability. Your notes don’t need to be queried or joined with anyone else’s notes.

redpoint• 2 years ago

This ^. This switch from managed to in house is a good example of only building when necessary.

altdataseller• 2 years ago

Also whether this data lake is worth the costs/effort. How does this data lake add value to the user experience? What is this “AI” stuff that this data lake enables?

For example, they mention search. But i imagine it is just searching only within your own docs. Which i presume should be fast and efficient if everything is sharded by user in Postgres.

The tech stuff is all fine and good, but if it adds no value, its just playing with technology for technology sakes

wearhere• 2 years ago

I too was surprised to read that they were syncing what reads, at a glance, to be their entire database into the data lake. IIUC the reason that Snowflake prioritizes inserts over updates is because you're supposed to stream events derived from your data, not the data itself.

mritchie7126• 2 years ago

They weren't that quiet about it:

Moving several large, crucial Postgres datasets (some of them tens of TB large) to data lake gave us a net savings of over a million dollars for 2022 and proportionally higher savings in 2023 and 2024.

patrickmay• 2 years ago

I'd like to see more details. 10s of TB isn't that large -- why so expensive?

methou• 2 years ago

Data lake Data warehouse

These aren't something I would like to hear if I'm still using Notion. It's very bold to publish something like this on their own website.

bnj2• 2 years ago

Could you expand on this?

lopkeny12ko2• 2 years ago

What's there to expand on? Do you not realize how bad of a look it is for a company to publicly admit, on their own blog, the amount of time and engineering effort they spent to package up, move, analyze, and sell all their customer's private data?

This is why laws like CCPA "do not sell my personal information" exist, which I certainly hope Notion is abiding by, otherwise they'll have lawyers knocking on their door soon.

bastawhiz6• 2 years ago

Those are just different words for "database". What do you care what kind of database your Notion data is sitting in?

TeMPOraL3• 2 years ago

A "data lake" strongly suggests there's lot of information the company needs to aggregate and process globally, which should very much not be the case with a semi-private rich notebook product.

zarmin• 2 years ago

Given how infuriating their implementation is of an in-app database, perhaps it's not that surprising.

wodenokoto• 2 years ago

That’s definitely something you want to do. Datalake can be home for raw and lightly refined data in an “analytics” database such as big query or just raw parquets. This is fast for large queries but slow for small queries. So you want refined data in a “regular” database like Postgres or mssql to serve all the dashboards.

DataDaemon• 2 years ago

OK, thanks, when E2EE ?

j45• 2 years ago

This was a nice read, interesting to see how far Postgres (largely alone) can get you.

Also we see how at self hosting within a startup can make perfect sense. :)

Devops that abstract away things in some cases to the cloud might just add to architectural and technical debt later, without the history of learning from working through the challenges

Still, it might have been a great opportunity to figure out offline first use of notion.

I have been forced to use anytype instead of notion for the offline first reason. Time to checkout to learn how they handle storage from the source code.

wejick• 2 years ago

I'm not familiar with S3 on datalake setup. When replicating a db table to S3, what format will be used?

And I'm wondering if it's possible to update the S3 files to reflect latest incoming changes on the db table?

mritchie7126• 2 years ago

The file format is often Parquet. The "table format" depend on what data lake you're using (e.g. Iceberg, Delta, etc.).

If you know Python, here's[0] a practical example of how Iceberg works.

0 - https://www.definite.app/blog/iceberg-query-engine

CyberDildonics2• 2 years ago

What's the difference between a data lake and a database with a filesystem?

anthomtb• 2 years ago

Add data warehouse to this list too.

OJFord• 2 years ago

Normalisation, relations, the filesystem, ..? I'm more curious about your view of a 'data lake' which is met by adding a filesystem to a db?

CyberDildonics2• 2 years ago

Normalisation, relations,

What does that mean?

syndicatedjelly• 2 years ago

I thought data lake was more of an advertising phrase, to show off how much data you have/can handle

Temporary_31337• 2 years ago

Web scale

crux• 2 years ago

Hi all—I'm the EM for the Search team at Notion, and I want to chime in to clear up one unfortunate misconception I've seen a few times in this thread.

Notion does not sell its users' data.

Instead, I want to expand on one of the first use-cases for the Notion data lake, which was by my team. This is an elaboration of the description in TFA under the heading "Use case support".

As is described there, Notion's block permissions are highly normalized at the source of truth. This is usually quite efficient and generally brings along all the benefits of normalization in application databases. However, we need to _denormalize_ all the permissions that relate to a specific document when we index it into our search index.

When we transactionally reindex a document "online", this is no problem. However, when we need to reindex an entire search cluster from scratch, loading every ancestor of each page in order to collect all of its permissions is far too expensive.

Thus, one of the primary needs that my team had from the new data lake is "tree traversal and permission data construction for each block". We rewrote our "offline" reindexer to read from the data lake instead of reading from RDS instances serving database snapshots. This allowed us to dramatically reduce the impact of iterating through every page when spinning up a new cluster (not to mention save a boatload in spinning up those ad-hoc RDS instances).

I hope this miniature deep dive gives a little bit more color on the uses of this data store—as it is emphatically _not_ to sell our users' data!

jzelinskie• 2 years ago

This is a fantastic post that explains a lot of the end product, but I'd love to hear more about the journey specifically on denormalizing permissions at Notion. Scaling out authorization logic like this is actually very under-documented in industry. Mind if I email you to chat?

Full disclosure: I'm a founder of authzed (W21), the company building SpiceDB, an open source project inspired by Google's internal scalable authorization system. We offer a product that streams changes to fully denormalized permissions for search engines to consume, but I'm not trying to pitch; you just don't often hear about other solutions built in this space!

infogulch2• 2 years ago

Interesting! Now I'm curious how you handle live permission changes and indexes with stale permission data.

jitl4• 2 years ago

(I’m not on the search team, but I did write some search stuff back in 2019, explanation may be outdated)

The blocks (pages are a block) in Notion are a big tree, with your workspace at the root. Some attributes of blocks affect the search index of their recursive children, like permissions: granting access to a page grants access to its recursive child blocks.

When you change permissions, we kick off an online recursive reindex job for that page and its recursive subpages. While the job is running, the index has stale entries with outdated permissions.

When you search, we query the index for pages matching your query that you have to. Because the index permissions can be stale, we also reload the result set from Postgres and apply our normal online server-side permission checks to filter out pages you lost access to but that have stale permissions in the index.

mritchie7126• 2 years ago

Hey! While you're here...

Iceberg and Delta Lake, on the other hand, weren’t optimized for our update-heavy workload when we considered them in 2022

Curious about your thoughts here. Have you followed Icebergs progress? Do you think it'd be a tougher decision in 2024 between Hudi and Iceberg?

atak1• 2 years ago

Curious - what do you guys use for the T step of your ELT? With nested blocks 12 layers deep, I can imagine it gets complicated to try to de-normalize using regular SQL.

Have you explored a pattern like https://runtrellis.com or https://unstructured.io/ for unnesting?

hobobaggins• 2 years ago

Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake.

Are they using this new data lake to train new AI models on?

Or has Notion signed a deal with another LLM provider to provide customer data as a source for training data?

gorgoiler• 2 years ago

It’s not a direct answer but from what Notion tell us about their own business:

* The team are based in the US, specifically California, and Notion Labs, Inc is a Delaware corporation.

* Their investment comes from Venture Capital and individual wealth. The investors are listed on Notion’s about page and are open about how they themselves became rich through VC funded tech companies.

There is a very open sense of panic in tech right now to climb to the top of the AI pile and not get crushed underneath. I would be amazed if there were any companies not enthralled by — and either already embracing or planning to embrace — the data-mining AI gold rush.

Notion is a great product but one would be naive to use it while also harboring concerns about data privacy.

jitl4• 2 years ago

(I work at Notion, but not on the data platform team)

We do not and will never sell customer data to anyone. We do not train AI models on customer data. As we state in our privacy policy for AI features (https://www.notion.so/notion/Notion-AI-Supplementary-Terms-f...):

Notion does not use your Customer Data or permit others to use your Customer Data to train the machine learning models used to provide Notion AI Writing Suite or Notion AI Q&A [added: our AI features]. Your use of Notion AI Writing Suite or Notion AI Q&A does not grant Notion any right or license to your Customer Data to train our machine learning models.

We do use various data infrastructure, including Postgres and the data lake, to index customer content both with traditional search infrastructure like Elasticsearch, as well as AI-based embedding search like Pinecone. We do this so you can search your own content when you're using Notion.

We wrote this article to explain how Notion's AI features works with customer data: https://www.notion.so/help/notion-ai-security-practices

jauntywundrkind• 2 years ago

Side-ish note, I really enjoyed a submission on Bufstream recently, a Kafka mq replacement. One of the things they mentioned is that they are working on building in Iceberg materialization, so Bufstream can automatically handle building a big analytics data lake out of incoming data. It feels like that could potentially tackle a bunch of the stack here. https://buf.build/blog/bufstream-kafka-lower-cost https://news.ycombinator.com/item?id=40919279

Versus what Notion is doing:

We ingest incrementally updated data from Postgres to Kafka using Debezium CDC connectors, then use Apache Hudi, an open-source data processing and storage framework, to write these updates from Kafka to S3.

Feels like it would work about the same with Bufstream, replacing both Kafka & Hudi. I've heard great things about Hudi but it does seem to have significantly less adoption so far.

alexliu518• 2 years ago

Thank you for the clarification! It's great to hear more about the efficient data management practices at Notion. Your team's innovative use of the data lake to streamline the reindexing process while ensuring user data privacy is impressive. Keep up the excellent work!

HermitX• 2 years ago

Great article, thank you for sharing! I have a question I’d like to discuss with the author. Spark SQL is a great product and works perfectly for batch processing tasks. However, for handling ad hoc query tasks or more interactive data analysis tasks, Spark SQL might have some performance issues. If you have such workloads, I suggest trying data lake query engines like Trino or StarRocks, which offer faster speeds and a better query experience.

jitl4• 2 years ago

(Notion employee)

AWS Athena packages Trino, I’ve been using it for some queries like “find all blocks that contain @-mentions”. It’s a great tool.

philippemnoel• 2 years ago

This is one of the best blog posts I've seen that showcase the UPDATE-heavy, "surface data lakes data to users" type of workload.

At ParadeDB, we're seeing more and more users want to maintain the Postgres interface while offloading data to S3 for cost and scalability reasons, which was the main reason behind the creation of pg_lakehouse.

Building and scaling Notion's data lake

Related Stories