relay23

or kafka, or any other publish/subscribe system

Sounds about right.

Kafka doesn't guarantee exactly once delivery at all, unless you're using Kafka Streams and even then your final output topic still won't get exactly once, the consumer group protocol doesn't allow for it.

It's cheaper until you get to a sizeable workload, and the P90+ latency is ridiculous.. the kafka api is weak and when you're not using kafka api you're limited on integration tools unless you want to be super locked in to GCP.

There is Redis Streams, but certainly not without it's problems. Super obscure, not a lot of client support.

Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.

> Rabbit can do everything Kafka does - and much more - in a more configurable manner. Sure, if you're doing like 10's of MB/s. RMQ is fast compared to AK if you're not adding durability, persistence, etc. Try to run gigabytes per second through it though, or stretch across regions, or meet RTO when the broker gets overloaded and crashes.. Get your shovel ready! ;)

Kafka itself is dumb but scalable and resilient, it's the client ecosystem that's massive compared to RabbitMQ. Count 10 stream processing, connectivity, ingestion or log harvesting platforms that use RMQ as it's backend, then name 10 languages that have supported libraries for RMQ.. then compare that to Kafka.

They always have.. you get it but you don't know it.

Maybe "it's enterprise" means that's what the enterprise standardized on. There are a couple of practical reasons that come to mind on why that's the case - a) it's more resilient and durable than messaging platforms, and b) it is a platform of dumb pipes, so to make it a central data bus managed by platform teams means that they don't have to get into the detail of which queues perform what functions, have what characteristics, etc. Rather the client teams in the various business units can take care of all of their "smarts" the way they want. It also handles log/telemetry ingestion, data platform integration, and interservice comms use cases which is pretty multi-functional. That's the primary reason why Kafka has become such a pervasive and common platform, it's not because it's trendy, in fact most operations teams would rather not even have to operate the kafka platform.

Also https://jepsen.io/analyses/redpanda-21.10.1

Are you sure performance would be acceptable if you just turned on fsync on every message?

Almost nobody auto commits offsets in real applications though. If you do then you should really stop :)