Chorus: A fast WAL for object storage

by cbrewster • 15 days ago

5 6 15 days ago

This was a really amazing post to read through. I can imagine so much cool stuff

I have plans to build a Network File System (an NFS server) using SlateDB and Chorus, and I’m excited to see what other people build as well.

Can you please elaborate more about what the benefits of combining SlateDB and chorus for NFS could be. I would love to know more in details

Also I would love to know what are some other use cases that you can think about it.

And from my understanding this is currently focused on GCP sides of it, do you think that it can be GCP agnostic as well if some other provider wishes to support for it, if so what are the things that would be needed by them to do so.

Also I really loved the writing, I feel as if these are the articles that I come for hackernews to read through. So creative!

A lot of it did go through my head though as I am not the most well-versed within this field but I really liked reading it and learning more about it to get more knowledge. (I also might like to know how does the pricing of the GCP rapid storage buckets compare to other things in general as well and which use-cases of chorus could justify its pricing/API costs as well.)

rockwotj3• 15 days ago

author here! Glad you liked the post :)

Motivation for combining Chorus and SlateDB for NFS

I work on an AI agent (tasklet) and we give every agent a linux machine. Having durable storage that is cheap, fast and multi-tenant is really important for our product. NFS is a great protocol (if complicated), and object storage is just the cheapest. But making it fast and reliable is key.

other use cases

Any use case for SlateDB that you are willing to pay more for less latency but keep disaggregated storage without another system.

GCP specific

Actually AWS and Azure zonal storage also support append operations, so I think the approach could be extended to all three major clouds. I don’t have a need for that ATM

pricing

Probably worth a whole separate blog TBH. It would be cheaper than Kafka but more expensive than just using the built in WAL for SlateDB or OSWALD

Imustaskforhelp3• 15 days ago

I work on an AI agent (tasklet) and we give every agent a linux machine. Having durable storage that is cheap, fast and multi-tenant is really important for our product. NFS is a great protocol (if complicated), and object storage is just the cheapest. But making it fast and reliable is key.

Wouldn't 9fs/plan9fs be more suitable for this use case from my limited understanding though?

Any use case for SlateDB that you are willing to pay more for less latency but keep disaggregated storage without another system.

I feel like IMO one of its benefits could be something like Figma/Google Docs alternative (Ironic considering that it would be running on google specific product but as you've mentioned in next point a potential support for AWS/Azure could also happen). I also found some use case within a live-streaming alternative in terms of how a live stream could add support for viewing things say a few seconds back. The current way seems to be much more messier than a simple s3. Though I feel like in any of these cases, pricing might be a factor.

I also find the use case that you shared really interesting as well. I wish you luck to see how that pans out and am looking forward for reading more technical pieces by you so keep publishing more and I am curious for more details as well.

There are so many interesting ideas when we combine s3 and databases. TigrisFS is taking an opposite approach to this by running filesystem on top of s3 on top of technically foundationDB whereas your approach/SlateDB essentially runs a DB on S3. It's fun to imagine both of these being used together and how both of these are approaching at somewhat of the same problem from completely different/opposite ways.

Chorus: A fast WAL for object storage

Related Stories