HN user
Making durable streams serverless
shikhar@s2.dev
OpenReplay is much more mature and full-featured, RePlaya is just the core session capture, listing, and replay functionality. OpenReplay has more dependencies, so self-hosting means running a full stack: Postgres, ClickHouse, Redis, and its backend services. RePlaya is one stateless Node process plus S2 (or self-hosted s2-lite).
Thanks! And agreed, session replays can be really useful to understand user behaviour such as product edge cases.
On cost, it's running the collector Node app (I'd expect a few $ per month at low volume), and the S2 stream backend.
If you use the S2 cloud service, cost is basically just the rrweb bytes. The rates are $0.075/GiB to write, $0.05/GiB-month to store, $0.10/GiB to read back over the internet. See s2.dev/pricing.md for an agent-friendly summary.
Assuming a typical few-minute session is ~1 MiB of events, ingesting it, storing it a month, and replaying it a couple of times (unlikely!):
1k sessions/mo ≈ $0.35
10k sessions/mo ≈ $3.50
100k sessions/mo ≈ $35
I agree that future seems inevitable. We work on plumbing for that, s2.dev - would be very happy to collaborate.
KV store in Rust, backed by a disaggregated, replicated journal https://github.com/s2-streamstore/s2-kv-demo
We run a copy using https://github.com/gabe565/ascii-movie, you can `nc starwars.s2.dev 23`
It's our favorite way of demoing s2.dev, https://x.com/jrdi/status/2014318511120670859
Personally I'd add an application level hash to protect the integrity of the records but that's just me.
The durability is for being able to replay the stream, a hash will not let you reconstruct the original message(s).
If you just need ephemeral comms, making it persistent is indeed overkill. But reliability challenges often come up with seemingly ephemeral comms too – think streaming responses from an LLM. The last mile can be pretty flaky e.g. iOS will cancel connections when users background an app. Using a durable stream for persisting the tokens means a client can ask to resume from where it left off / from the beginning of the stream, and the data would be available without having to re-inference.
Yes, that is a reasonable way to think about it! And as s2-lite is designed as a single-node system, there is a natural source of truth on what the latest records are for consuming in real-time.
Can this be used as an embedded lib instead of a separate binary as an API?
Did not architect explicitly for that, but should be viable. You could use the `Backend` directly, is what the REST handlers call https://docs.rs/s2-lite/latest/s2_lite/backend/struct.Backen...
Happy to accept contributions that make this more ergonomic.
And am I understanding correctly that if I pointed 2 running instances of s2-lite at the same place in s3 there would be problems since slatedb is single writer?
SL8 will fence the older writer, thanks to S3 conditional writes. I think there would be potential for stale reads until the fencing happens...
ED: Fresh discussion in https://discord.com/channels/1232385660460204122/12323856609...
The stale read potential can be mitigated, https://github.com/s2-streamstore/s2/issues/91
Very cool! Looking forward to catching up. Also checkout https://s2.dev/blog/kv-store
We wanted S2 to be one API. Started out with gRPC, added REST - then realized REST is what is absolutely essential and what most folks care about. gRPC did give us bi-directional streaming for append/read sessions, so we added that as an optional enhancement to the corresponding POST/GET data plane endpoints (the S2S "S2-Session" spec I linked to above). A nice side win is that the stream resource is known from the requested URL rather than having to wait for the first gRPC message.
gRPC ecosystem is also not very uniform despite its popularity, comes with bloat, is a bit of a mess in Python. I'm hoping QUIC enables a viable gRPC alternative to emerge.
Is this for like "making your own Twitch" or something, where streams have to scale to thousands-to-millions of consumers?
Yes, this can be a good building block for broadcasting data streams.
s2-lite is single node, so to scale to that level, you'd need to add some CDN-ing on top.
s2.dev is the elastic cloud service, and it supports high fanout reads using Cachey (https://www.reddit.com/r/databasedevelopment/comments/1nh1go...)
This is fair question. A stream here == a log. Every write with S2 implementations is durable before it is acknowledged, and it can be consumed in real-time or replayed from any position by multiple readers. The stream is at the granularity of discrete records, rather than a byte stream (although you can certainly layer either over the other).
ED: no k8s required for s2-lite, it is just a singe binary. It was an architectural note about our cloud service.
It seems SL8 supports writing the WAL to local disk already https://github.com/slatedb/slatedb/issues/162
Will look into how to enable that option from s2-lite
Shoutout to CodesInChaos for suggesting that instead of a mere emulator, should have an actually durable open source implementation – that is what we ended up building with s2-lite! https://news.ycombinator.com/item?id=42487592
And it has the durability of object storage rather than just local. SlateDB actually lets you also use local FS, will experiment with plumbing up the full range of options - right now it's just in-memory or S3-compatible bucket.
So I'd try so share as much of the frontend code (e.g. the GRPC and REST handlers) as possible between these.
Right on, this is indeed the case. The OpenAPI spec is also now generated off the REST handlers from s2-lite. We are getting rid of gRPC, s2-lite only supports the REST API (+ gRPC-like session protocol over HTTP/2: https://s2.dev/docs/api/records/overview#s2s-spec)
Very cool! Would you consider making the streaming backend pluggable? s2.dev could make a lot of sense as a serverless option, and a self-hostable OSS implementation of the API is also coming soon. S2 is great for agent session-level streams (https://s2.dev/blog/agent-sessions), and unlike Redis Streams all data is always completely durable on object storage.
It seemed like the kind of Jepsen outcome where folks would be considering alternatives, but yeah maybe it was not appropriate to plug here.
The cloud offering is self-serve, no need to get on a call at all. An open source, self-hosted option is in progress https://github.com/s2-streamstore/s2?tab=readme-ov-file#s2-l...
We do deterministic simulation testing
https://s2.dev/blog/dst https://s2.dev/blog/linearizability
We have also adopted Antithesis for a more thorough DST environment, and plan to do more with it.
One day we will engage Kyle to Jepsen, too. I'm not sure when though.
If you are looking for a serverless alternative to JetStream, check out https://s2.dev
Pros: unlimited streams with the durability of object storage – JetStream can only do a few K topics
Cons: no consumer groups yet, it's on the agenda
Postgres is a way better fit than Kafka if you want a large number of durable streams. But a flexible OLTP database like PG is bound to require more resources and polling loops (not even long poll!) are not a great answer for following live updates.
Plug: If you need granular, durable streams in a serverless context, check out s2.dev
multiplayer wall (iykyk) for HN: https://s2.dev/demos/y-s2?room=neon-canyon-7857
Yes, currently it has its own /fetch endpoint that then makes S3 GET(s) internally. One potential gotcha depending on how you are using it, an exact byte "Range" header is always required so that the request can be mapped to page-aligned byte range requests on the S3 object. But with that constraint, it is feasible to add an S3 shim.
It is also possible to stop requiring the header, but I think it would complicate the design around coalescing reads – the layer above foyer would have to track concurrent requests to the same object.
Essentially CacheLib in Rust
foyer draws inspiration from Facebook/CacheLib, a highly-regarded hybrid cache library written in C++, and ben-manes/caffeine, a popular Java caching library, among other projects.
Foyer is a great open source contribution from RisingWave
We built a S3 read-through cache service for s2.dev so that multiple clients could share a Foyer hybrid cache with key affinity, https://github.com/s2-streamstore/cachey
Hi mertletee, I'd like to understand the request better, mind dropping me an email? It's in my profile
Check out ZeroFS (https://www.zerofs.net), which is using SlateDB (https://slatedb.io/)
ED: Now I catch your drift, it would indeed be cool. ZeroFS requires a commitment to the SlateDB LSM data format.
Yes, that's how we are running it at s2.dev, auto-scaled per-AZ deployments. https://www.reddit.com/r/databasedevelopment/comments/1nh1go...
Very cool! Maybe you'll consider turning it distributed by using s2.dev for the append-only event logs :)
Someone tried this with XTDB, https://github.com/chucklehead-dev/s2-log
How we run it:
Auto-scaled Kubernetes deployments, one for each availability zone, currently on m*gd instances which give us local NVMe. The pods are able to easily push GiBps with 1-2 CPUs used — network is the bottleneck so we made it a scaling dimension (thanks KEDA).
On the client side, each gateway process uses kube.rs to watch ready endpoints in the same zone as itself, and frequently polls /stats exposed by Cachey for recent network throughput as a load signal.
To improve hit rates with key affinity, clients use rendezvous hashing for picking a node, with bounded load (https://arxiv.org/abs/1608.01350) – if a node exceeds a predetermined throughput limit, the next choice for the key is picked.
We may move towards consistent hashing – it would be a great problem to have, if we needed so many Cachey pods in a zone that O(n) hashing was meaningful overhead! An advantage with the current approach is it does not suffer from the cascaded overflow problem (https://arxiv.org/abs/1908.08762).