How we saved $100k a month with Kafka networking optimization

https://segment.com/blog/kafka-optimization

by fullung • 6 years ago

11 3 6 years ago

Given the way you were able to logically group data so that producers and their consumers are in the same zone, would it not have been simpler to just create three completely independent 3-node clusters, rather than worry about all this at the partition and keying level?

ir123• 6 years ago

Yes this seems confusing to me as well. AZs are usually used for reliability. I'm looking at their new topology and wondering why AZs were used at all?

Maybe if one AZ fails only 1/3 of the producer, topics and consumers fail. Seems like a lot of overhead for a relatively minor advantage.

pryz• 6 years ago

The idea is that we are running few multi-tenant Kafka clusters to amortize the operation cost. These clusters host many topics with different topology depending on different requirements, cost profile and scale.

Some clusters are hosting hundreds of topics, thousands of partitions and used (consumers/producers) by hundreds of services.

How we saved $100k a month with Kafka networking optimization

Related Stories