Ursa – a new Iceberg-first storage engine for Kafka

https://topicpartition.io/blog/ursa-a-new-lakehouse-first-storage-engine-for-kafka

by enether • 3 months ago

3 2 3 months ago

Ursa has been around for a while (I first heard about it last year), but it stood out to me among the many diskless Kafka implementations because it actually treats Iceberg as first class, i.e. Iceberg is the storage. Other implementations like AutoMQ, RedPanda, and WarpStream treats Iceberg as tiered storage, i.e. you write to Kafka, Kafka writes to disk and ack, then Kafka will asynchronously write to Iceberg in the background. However it also means that Ursa cannot be a drop-in replacement of Kafka, especially if you need milliseconds end-to-end latency (from producer to consumer), since cloud storage + Iceberg adds a lot of latency.

But from my experience, I'd argue 90% of use cases of Kafka don't need sub-100ms latency. In my previous company (fintech company), pretty much 99% use cases are for asynchronous processing that can tolerate seconds of latency. We just use Kafka because it's there, there's a team that operates it for everyone so the ugly parts are hidden from everyone else, and the cost is pretty much "subsidized" (the cost is attributed to the team that operates it).

enether• 3 months ago

Yeah, which is why this Ursa-for-Kafka (UFK) fork can be a drop-in replacement. It holds both the classic Kafka topics with disks, and the Ursa engine topics that have higher latency.

I agree with your assessment re: latency. I've been very vocal about this ever since this type of architecture came out.

Ursa – a new Iceberg-first storage engine for Kafka

Related Stories