Show HN: Baerly-storage, a document DB that runs per request, no DB server

by baerbaerbaer • 16 days ago

18 1 16 days ago

Hi HN, Eric here. I did a thing, and I'm super excited to share. By leaning on S3's strong read-after-write consistency, I made a datastore without a persistent database server. As in no binary and no daemon. Just your app and a bucket.

baerly-storage is a document database with no separate database process: the read/write machinery runs to completion inside your request handler, and the durable state lives in S3/R2.

The load-bearing trick: for each collection, a write commits by creating the next numbered log object (`log/<seq>.json`) with `If-None-Match: ""` create-if-absent. That one conditional create is* the commit. `current.json` is not the latest-commit pointer; it is compaction state plus hints. Readers load the snapshot it names and then forward-probe the dense numbered log by GET until the first missing sequence.

If two writers race, exactly one wins a sequence. If a writer sees the slot occupied, it reads the occupant back: same session, same seq, same full entry means "my success response was lost"; anything else means a peer won and this writer retries at the next empty slot. New index markers are written before the log create, stale markers after, so a crash can leave extra index candidates, but should not hide committed rows. Reads stay pure; bounded compaction/GC rides successful writes.

The TypeScript package is the first implementation. The actual contract is the object layout and protocol: content-addressed document bodies, a dense numbered log, snapshot objects, advisory index markers, and `current.json` as compaction state rather than commit authority. Another implementation could speak the same protocol if it honors the same conditional-write and tail-probe rules.

I built this at Gusto because we're seeing a lot of AI-authored internal tools like dashboards, trackers, one-off Claude artifacts that are real enough to need shared durable state but not big enough to justify Postgres + Docker + secrets + on-call and all the ceremony that implies.

The framing that stuck for me was that you need three primitives to build software these days: compute, tokens, and storage. For these workloads, compute has an answer (FaaS, scale to zero), and tokens are just an API call. Storage was missing one.

The cool thing is that this only became possible recently. S3 shipped strong read-after-write in late 2020 and create-if-absent conditional writes in Aug 2024. It's the same "write immutable data, publish with a small atomic step" playbook as Iceberg / Delta / Litestream, but the high-frequency coordination point is the log append itself, so there's no catalog service or lock table.

I did a bunch of conformance and benchmark testing and, TBH, this thing scales better than I expected. Something like 30 requests per sec sustained is still reasonable on both perf and cost (I have a whole writeup in the docs). But, obviously, this has constraints. It gives you per-collection linearizability but no cross-collection transactions or joins, so the best fit test is "can this app's most important screen come from one collection?" It's read-cost-heavy at scale (every read folds a log tail), and at high write volume, the S3 bill from Class A operations starts to price you out.

But also, that's the graduation signal, and I'm considering graduation a success case. I made a CLI to give a clean SQL exit: `baerly export --target=postgres`.

Oh, and as a bonus, the API treats LLMs as a first-class consumer, and part of my test suite was to zero-shot ~7 different kinds of apps with a Sonnet-class model. The whole public API is 8 verbs and 6 predicate operators. All typesafe, with a 12k-token surface that is small enough to fit in context.

Happy to go deep on the consistency model, which was the hard part.

Show HN: Baerly-storage, a document DB that runs per request, no DB server

Related Stories