wanshao

Scaling Kafka clusters has always been a challenging task. Kafka uses the ISR multi-replica mechanism to ensure data persistence, which was a natural choice when the cloud was not yet mature. However, in 2024, when cloud computing is very mature, this design seems a bit outdated. When the cluster is scaled, we must move the data of the partitions. Moving partition data will affect normal reading and writing, and this process may last a long time, tens of minutes or a few hours, depending on the amount of your data. This means that often, even when the business scale has expanded to the critical point, we still dare not carry out such operations as expansion, because there is a high execution risk.

AutoMQ is fully aware of the root cause of this problem, so it has redesigned and implemented the entire storage layer of Kafka based on the cloud (we call ourselves cloud-first Kafka). Offload data persistence to cloud storage, and ensure data persistence by the multi-replica mechanism inside cloud storage. This also gives us the ability to build a more powerful self-balancing ability than Cruise Control. This blog post details how we technically achieve this, and we hope it can bring some new insights to everyone.

In recent years, we have seen many challengers to Kafka emerge. Without exception, they all claim to be fully compatible with the Kafka API. Some challengers have even directly forked Kafka's code to innovate further. Has the Kafka API now become the de facto standard in the stream processing domain? How has it achieved this status?

What is BufStream?Is it a new streaming system?

Thanks for your advice

Thanks, I've joined.

I agree with your viewpoint. The crux of the matter is not whether to use tiered storage or not, but what trade-offs have been made in the specific storage architecture and what benefits have been gained. Here(https://github.com/AutoMQ/automq?tab=readme-ov-file#-automq-...) is a qualitative comparison chart of streaming systems including kafka/confluent/redpanda/warpstream/automq. This comparison chart does not have specific numerical comparisons, but purely based on their trade-offs at the storage level, I think this will be of some use to you.

Buddy, you've hit the nail on the head. Everything is a trade-off. For a stream processing system, I believe it's entirely possible to balance cost, ease of use, and latency. AutoMQ(https://github.com/AutoMQ/automq) is also a stream system built on top of S3. Its storage scheme introduces a very small size of EBS storage as a persistent write cache, and then asynchronously compacts the memory data to S3, taking into account latency while retaining the advantages of warpstream. Tiered-storage is just a form, how to implement it depends on you.

Examples of this are quite common. Many tech company blog pages offer tag-based searches. The articles published can be tagged either by the system or manually, allowing us to filter content we're interested in by searching for these tags.

That makes sense. Every choice has its pros and cons. Simplicity is the philosophy of HN. If complexity gets out of control, HN would cease to exist. However, perhaps some small experiments could be feasible while maintaining simplicity?

Thank you for sharing. However, I might be looking for more specific categorizations. For instance, if I'm only interested in AWS-related content, I would like to be able to search for the 'AWS' tag.

The last conversational approach is very practical because it provides a concrete example.

This is a tremendous feat. Your success should be greatly tied to the actions you've taken. You've listed a lot of experiences that contributed to this outcome. Many projects have done the same, but they haven't succeeded. What I'm curious about is, among the suggestions you've listed, which measure has contributed the most to this result?

The ban is not just about the issue of traffic fabrication. The fake story created by this influencer has occupied limited social media resources. This has prevented many more important news and events from being understood by the public.

Thank you for your response. I will adjust my behavior to ensure that I comply with community rules and provide something truly valuable to everyone.

Thank you for your explanation and advice. I will adjust my behavior to ensure that I can comply with the rules of the HN community.

The elasticity of the public cloud mainly manifests in the user's ability to use its resources on demand. When you need more resources to support the workload, you can quickly purchase resources; when you no longer need these resources, you can release them immediately. The elasticity of the public cloud mentioned in the article mainly refers to its ability to provide resources to users on demand.

Thanks for your kind action.I want to know is the post really dead? When I look at it in my personal account's submissions, I find that the post has not been marked as dead. My account has been set to showdead=yes. If it's really as you said, does this mean that I can't see the dead tag myself?"

It's so strange, now others can see my post as well. The title of the post is: 'Why it's difficult to utilize the elasticity of the public cloud?'. When I encountered the issue, I also sent an email to the moderator explaining the situation. I wonder if it has anything to do with their intervention. Does Hacker News have a content review process?

When Kafka deals with large volumes of data, scaling out or in requires large data replication of the partitions, which is a very heavy operation. With the architecture based on S3 using shared storage, everything becomes much simpler. In my view, the trade-off is that we additionally use EBS as a persistent write cache to work with S3, which to some extent eliminates the latency issue of accessing S3.

Thank you for your openness to new things. Your support is indeed very important to a new project. BTW, AutoMQ itself provides real comparison data tested on AWS, and the related code is open for users to reproduce these tests. For a cost comparison with Apache Kafka, you can refer to: https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aw.... If you care about performance data, you can refer to our performance white paper: https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkx....

jpgvm is right. AutoMQ didn't modify the computation layer of Kafka, but only revamped the storage layer. This means you can use AutoMQ just like Kafka. This is very user-friendly for those who are already using Kafka and applying tools within the Kafka ecosystem.

Reasonable skepticism. Rather than changing the agreement at some point in the future, it's better to clarify it from the beginning. Perhaps introducing an enhanced version of the BSD license, which does not allow the project to modify the license at any point in the future, would be a good idea.