Horizontally scaling Kafka consumers with rendezvous hashing

https://www.tinybird.co/blog-posts/kafka-horizontal-scaling

by luismedel • 3 years ago

12 3 3 years ago

__all__• 3 years ago

The idea is nice but requires trusting their users a lot. How do they prevent users from setting super high "linker_workers" to be consumed by every agent? This could open the door for malicious users to saturate the entire system...

_peregrine_• 3 years ago

I'm pretty sure linker_workers is something that they configure, I don't think it's exposed to the user to configure.

alejandromav• 3 years ago

Hi! Alex here, Product Manager at Tinybird.

_peregrine_ is right, we do have control over the linker_workers value for every agent. Agents are not exposed to our users, everything is transparent for them.

As a user, you just need to create a new connection to your Kafka cluster providing your credentials, and then choose the topic you want to ingest into Tinybird. We take care of everything for your, we balance the worker using the approach explained in the blog post.

We can also fine tune it for specific users, or relevant events such as Black Friday. It's a bit different for Enterprise customers, in those cases we can set up dedicated agents and fine tune linker_workers and other parameters to optimize for their use case.

If you have any further questions you can reach us on our public Slack channel or via email.

Thanks!

Horizontally scaling Kafka consumers with rendezvous hashing

Related Stories