Why was Apache Kafka created?

https://bigdata.2minutestreaming.com/p/why-was-apache-kafka-created

by enether • 11 months ago

201 218 11 months ago

bigdata.2minutestreaming.com

blast2• 11 months ago

(I was wondering if this was some sort of generated ripoff but the author worked on Kafka for 6 years: https://x.com/BdKozlovski.)

enether7• 11 months ago

What do you mean by "generated ripoff"? Are you saying it read like AI?

blast2• 11 months ago

I mean that it looked like either AI generated or blogspam or both. Not the writing though, more the look of the page. Happy to be wrong.

polynomial• 11 months ago

Why was it named that is also a question.

isaacremuant• 11 months ago

Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work.

From Wikipedia.

atombender7• 11 months ago

A system optimized for writing could also describe the machine in Kafka's "In the Penal Colony".

clippy992• 11 months ago

Kafkaesque to configure and get running for simple tasks.

physicles3• 11 months ago

That’s funny, I assumed it was called Kafka because the act of processing items in a queue over and over could be described as kafkaesque.

theyinwhy• 11 months ago

It is called Kafka because it can write.

throw3108222• 11 months ago

It's a bit like calling a dictation software "Hitler" because he also liked to dictate.

kachapopopow3• 11 months ago

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

mrweasel• 11 months ago

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but you are making things harder than they needed to be.

adev_3• 11 months ago

Why they'd need Kafka was always our first question, never got a good answer from a single one of them

"To follow the hype train, Bro" is often the real answer.

If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.

Or just freaking MQTT.

MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).

slau• 11 months ago

ZMQ is not a managed queue. It’s networking library.

Joeri• 11 months ago

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA.

If you need all those things, there just are not a lot of options.

EdwardDiego4• 11 months ago

I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?"

The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.

Using S3 was better, but we still hit data loss problems (on our end, not S3 's, to clear) which isn't great when you need to bill on some of that data.

Then I heard about Kafka, bought a copy of I <3 Logs, and decided that maybe Kafka was worth the complexity, and boom, it was. No more data loss, and a happier business management.

I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."

But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.

And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.

Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."

selkin• 11 months ago

KIP-932[0] adds queue semantics for Kafka. You may still want to use another queue though: as always, no one size fits all.

[0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

kachapopopow3• 11 months ago

I do not recommend Redis (janky implementation, subscribers drop randomly, java libraries are a crime against humanity) or RabbitMQ (memory issues), ZMQ is not a messaging queue, named pipes are not reliable and what the hell is SQS.

HarHarVeryFunny5• 11 months ago

IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case.

I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.

In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.

GoblinSlayer• 11 months ago

I think our salesmen were happier to sell kafka, because it's enterprisey. Redis is better? Well, now we use kafka and redis.

rcombatwombat2• 11 months ago

Let's not forget NATS.io which does queuing (and a whole lot of other things from at least once low latency pub/sub to request/reply, streaming and all the way to KV and Object store) very well and is very light weight to run and much simpler to run than Kafka.

AtlasBarfed5• 11 months ago

If you're running a distributed system...

You're running a distributed system. They aren't simple.

Especially on AWS. AWS is really a double-bladed sword. Yeah, you'll get tutorials to set up whatever distributed system pretty quickly, but your nodes aren't nearly as reliable. Your networking isn't nearly as reliable. Your costs aren't nearly as reliable and administration. Headaches go up in the long run

SJC_Hacker2• 11 months ago

This is what alot of people don't get. They think Kubernetes is a solution to everything. In reality all k8s is makes the mechanics a little easier.

But if you don't understand distributed systems, it almost makes it worse because its tempting to segment the system across dozens of microservices which all have to talk with each other and synchronize, and the whole thing becomes a buggy, slow clusterfuck.

shikhar3• 11 months ago

You might like what we are building with https://s2.dev :)

zbentley• 11 months ago

Looks interesting; does it take a different architectural approach than WarpStream did?

That’s not coded “you’re reinventing the wheel”; WarpStream had some significant drawbacks, so I’m truly curious about different approaches in the message-log-backed-by-blob-store space.

raverbashing• 11 months ago

With Kafka, at least the naming is on point

clippy992• 11 months ago

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

oulipo2• 11 months ago

Have you tried Redpanda?

majormajor4• 11 months ago

If you are using this sort of redis queue (https://redis.io/glossary/redis-queue/) with PUSH/POP vs fan-out you're working on a very different sort of problem than what Kafka is built for.

Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java.

However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks.

But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).

njitbew• 11 months ago

and it feels bloated (Java!)

I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

slipperydippery3• 11 months ago

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle.

Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years.

Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.

poemxo2• 11 months ago

Starting up a Java program takes much longer than it should and that affects perception.

buckle80172• 11 months ago

Java the language and Java the runtime are fine.

The way most Java code is written is terrible Enterprise factory factory factory.

rvz3• 11 months ago

I'm curious, what exactly feels bloated about Java?

Everything.

Why do you think Kubernetes is NOT written in Java?

cameronh90• 11 months ago

what exactly feels bloated about Java?

https://docs.spring.io/spring-framework/docs/2.5.x/javadoc-a...

burnt-resistor• 11 months ago

Have you ever run on-prem Atlassian products or any enterprise JVM apps?

They hog RAM, are slow, and are a bitch to configure.

cortesoft• 11 months ago

It doesn't have to be 'hyper-scaled' to be needed, unless we have widely different definitions of hyper scale. Access logs from a few thousand servers with medium traffic will push you past any single instance service, and Kafka works great for that workload.

rvz3• 11 months ago

Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today.

Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.

AtlasBarfed5• 11 months ago

I thought Kafka ditched the zookeeper

closeparen• 11 months ago

Publishing an event to Kafka puts it “out there” in a way that guarantees it won’t be lost and allows any number of interested consumers, including the data warehouse, to deal with it at their leisure (subject to retention period which is typically like 72h). For us, your Kafka topics and their schemas are as much a part of your API as your gRPC IDLs. Something like Redis or 0MQ feels more appropriate for internal coordination between instances of the same service, or at least a producer that has a specific consumer in mind.

haddr• 11 months ago

Couldn’t disagree more… if you go the ZMQ you are left alone handling many things you get in Kafka for free. If you have any sort of big data problems then good luck. You are going to reinvent the wheel.

gf00012• 11 months ago

If you have this stance to technology ("feels bloated", meme-level knowledge on Java), then I really hope that you are not responsible for technical decisions.

zug_zug2• 11 months ago

My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.

majormajor4• 11 months ago

What off the shelf tools in 2012 would you propose, exactly?

tomrod• 11 months ago

Sounds like MQTT?

zug_zug2• 11 months ago

Make it less event-orchestrated and use a db. It’s just a social network for recruiters it’s not as complicated as they like to pretend.

You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.

jyounker• 11 months ago

And what would that off-the-shelf software have been?

dktalks• 11 months ago

I worked on this while I was at LI and I think the major selling point back then was Replayability of messages but it was something similar to what you would get with Pub/Sub. We could also have multiple clients listening and processing same messages for their own purposes so you could use the same queue and have different clients process them as they wanted.

pojzon• 11 months ago

Its the ability to replay messages at later notice when needed.

At least this was the reason we decided to use Kafka instead of simple queues.

It was useful when we built new consumer types for the same data we already processed or we knew we gonna have later but cant build now due to prorities.

varbhat• 11 months ago

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

nchmy3• 11 months ago

I dont have kafka experience, but nats is absolutely amazing. Just a complete pleasure to use, in every way.

https://www.synadia.com/blog/nats-and-kafka-compared

dijit3• 11 months ago

I got really pissed off with their field CTO for essentially trying to pull the wool over my eyes regarding performance and reliability.

Essentially their base product (NATs) has a lot of performance but trades it off for reliability. So they add Jetstream to NATs to get reliability, but use the performance numbers of pure NATs.

I got burned by MongoDB for doing this to me, I won’t work with any technology that is marketed in such a disingenuous way again.

AtlasBarfed5• 11 months ago

Don't implement any distributive technology until aphyr has put it through the paces, and even then... Pilot

nchmy3• 11 months ago

You mean Jetstream?

Can you point to where they are using core NATS numbers to describe Jetstream?

sea-gold• 11 months ago

There is a good comparison between NATS, Kakfa, and others here: https://docs.nats.io/nats-concepts/overview/compare-nats

rockwotj• 11 months ago

Maybe needs a neutral party comparison :)

The delivery guarantees section alone doesn’t make me trust it. You can do at least once or at most once with kafka. Exactly once is mostly a lie, it depends on the downstream system: unless going back to the same system, the best you can do is at least once with idempotancy

chatmasta2• 11 months ago

It’s on the NATS website and “nats” appears in the URL three times, so maybe this isn’t the most objective source.

atombender7• 11 months ago

NATS is very good. It's important to distinguish between core NATS and Jetstream, however.

Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No durability, so when a consumer disconnects, it will miss messages sent in its absence. But this means it's very lightweight. Subjects are just wildcard paths, so you can have billions of them, which means RPC is trivial: Send out a message and tell the receiver to post a reply to a randomly generated subject, then listen to that subject for the answer.

NATS organizes brokers into clusters, and clusters can form hub/spoke topologies where messages are routed between clusters by interest, so it's very scalable; if your cluster doesn't scale to the number of consumers, you can add another cluster that consumes the first cluster, and now you have two hubs/spokes. In short, NATS is a great "message router". You can build all sorts of semantics on top of it: RPC, cache invalidation channels, "actor" style processes, traditional pub/sub, logging, the sky is the limit.

Jetstream is a different technology that is built on NATS. With Jetstream, you can create streams, which are ordered sequences of messages. A stream is durable and can have settings like maximum retention by age and size. Streams are replicated, with each stream being a Raft group. Consumers follow from a position. In many ways it's like Kafka and Redpanda, but "on steroids", superficially similar but just a lot richer.

For example, Kafka is very strict about the topic being a sequence of messages that must be consumed exactly sequentially. If the client wants to subscribe to a subset of events, it must either filter client-side, or you have some intermediary that filters and writes to a topic that the consumer then consumes. With NATS, you can ask the server to filter.

Unlike Kafka, you can also nack messages; the server keeps track of what consumers have seen. Nacking means you lose ordering, as the nacked messages come back later. Jetstream also supports a Kafka-like strictly ordered mode. Unlike Kafka, clients can choose the routing behaviour, including worker style routing and deterministic partitioning.

Unlike Kafka's rigid networking model (consumers are assigned partitions and they consume the topic and that's it), as with NATS, you can set up complex topologies where streams get gatewayed and replicated. For example, you can streams in multiple regions, with replication, so that consumers only need to connect to the local region's hub.

While NATS/Jetstream has a lot of flexibility, I feel like they've compromised a bit on performance and scalability. Jetstream clusters don't scale to many servers (they recommend max 3, I think) and large numbers of consumers can make the server run really hot. I would also say that they made a mistake adopting nacking into the consuming model. The big simplification Kafka makes is that topics are strictly sequential, both for producing and consuming. This keeps the server simpler and forces the client to deal with unprocessable messages. Jetstream doesn't allow durable consumers to be strictly ordered; what the SDK calls an "ordered consumer" is just an ephemeral consumer. Furthermore, ephemeral consumers don't really exist. Every consumer will create server-side state. In our testing, we found that having more than a few thousand consumers is a really bad idea. (The newest SDK now offers a "direct fetch" API where you can consume a stream by position without registering a server-side consumer, but I've not yet tried it.)

Lastly, the mechanics of the server replication and connectivity is rather mysterious, and it's hard to understand when something goes wrong. And with all the different concepts — leaf nodes, leaf clusters, replicas, mirrors, clusters, gateways, accounts, domains, and so on — it's not easy to understand the best way to design a topology. The Kafka network model, by comparison, is very simple and straightforward, even if it's a lot less flexible. With Kafka, you can still build hub/spoke topologies yourself by reading from topics and writing to other topics, and while it's something you need to set up yourself, it's less magical, and easier to control and understand.

Where I work, we have used NATS extensively with great success. We also adopted Jetstream for some applications, but we've soured on it a bit, for the above reasons, and now use Redpanda (which is Kafka-compatible) instead. I still think JS is a great fit for certain types of apps, but I would definitely evaluate the requirements carefully first. Jetstream is different enough that it's definitely not just a "better Kafka".

shikhar3• 11 months ago

Jetstream clusters don't scale to many servers (they recommend max 3, I think)

Jetstream is even more limited than most Kafkas on number of streams https://github.com/nats-io/nats-server/discussions/5128#disc...

pdimitar3• 11 months ago

That's an amazing analysis, thank you so much!

What are your impressions of Red panda?

We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs.

Interested if Redpanda is doing better than either.

jnmatsynadia4• 11 months ago

Thanks for appreciating NATS.io and your comment!

Jetstream clusters don't scale to many servers (they recommend max 3, I think)

You can have clusters with many servers in them, 3 is actually the minimum required if you want fault-tolerance, that's how you scale JetStream horizontally: you let the stream (which can be replicated 1, 3 or 5 times) spread themselves over the servers in the cluster.

JetStream consumers create state on the server (or servers if they are replicated) and are either durable with a well known name or 'auto-cleanup after idle time' (ephemeral), and indeed allow you to ack/nack each message individually rather than just having an offset.

However, that is with the exception of 'ordered consumers', which are really the closest equivalent to Kafka consumers in that the state is kept in the client library rather than on the servers. They deliver the message in order to the client code, they take care of re-deliveries and recovering from things like getting disconnected from a server, and only ever deliver messages in order, no need to explicitly ack the messages, and if you want to persist your offset (which is the sequence number of the last message the ordered consumer delivered) just like the consumer group clients in Kafka persist their offset in a stream, you would persist your offset in a NATS KV bucket.

And indeed you can now even go further and use batched direct gets to get very good read speed from the stream and no extra server state in the server besides an entry in the offset KV, performance of the batched direct gets is very high and can match the ordered consumer's speed. Besides no incurring no server state, another advantage of stateless consuming is that all the servers replicating the stream will be used to process direct get requests not just the currently elected leader (don't forget to enable direct gets for the stream, it's not on by default). So you can scale the read throughput horizontally by increasing the number of replicas.

The mechanics of replication: streams and stateful consumers can be replicated using 1, 3 or 5 servers. Servers connect directly together to form a cluster and jetstream assets (streams/consumers) are spread out over the servers in the cluster. Clusters can be connected together for form super-clusters. Super-cluster means that access to JetStream assets is transparent: streams/consumers located in one cluster can be accessed from any other cluster. You can have streams that mirror or source from other streams, those mirrors could be located in other clusters to offer faster local access. You can easily move on the fly JS assets from one cluster to another. Leaf nodes are independent servers (which can be clustered) that connect to a cluster like a client would. Being independent means they have their own security for their own clients to connect to them, they can have their own JS domain and you can source to/from streams between the leaf node's domain and the hub (super-cluster). Leaf nodes can be daisy chained.

BiraIgnacio• 11 months ago

It was created to teach me the concept of love-hate relationships

fifilura2• 11 months ago

I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well.

The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.

Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.

Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.

I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?

At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.

physicles3• 11 months ago

Totally agree with this. I’ll add that replaying your data needs special tooling to 1) find the correct offsets on each topic, and 2) spin up whatever daemon will consume that data out-of-band from normal processing, and shut it down when completed.

I don’t remember where I read this, but someone made the observation that writing a stream processing system is about 3x harder than writing a batch system, exactly for all the reasons you mentioned. I’m looking at replacing some of our Kafka usage with a clickhouse table that’s ordered and partitioned by insertion time, because if I want to do stuff with that data stream, at least I can do a damn SQL query.

bjourne• 11 months ago

I read it and looked at the block diagrams. I still don't get it. You have "data integration problems". Many software share data. Use database. Problems solved.

erulabs3• 11 months ago

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale. If you bristle at that, know that using read-replicas of your postgres database present the same correctness problems.

These days though, unless I was at Fortune 100 scale, I'd absolutely turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run.

Also I like Kafka because I met two pretty Russian girls in San Francisco a decade back and the group we were in played a game where we described what the company we worked for did in the abstract, and then tried to guess the startup. They said "we write distributed streaming software", I guessed "confluent" immediately. At the time confluent was quite new and small. Fun night. Fun era.

betaby• 11 months ago

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great.

I sue Kafka precisely for that. Redis Cluster Streams have AOF persistence logs as I see from the doc. How stable it is?

dxxvi• 11 months ago

turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run

I don't have any experience with Redis Cluster Streams. Could you please tell us how it is simpler to manage? IMO, installing and managing a Kafka cluster in a non Fortune 100 scale is simple enough: run 1 java command for zookeeper, run another java command for a broker (with recent version of Kafka, zookeeper is not needed anymore). The configuration files are not very simple but not very complicated either. When we have another machine, we can run another broker on it.

Redis Cluster Streams is cheaper to run because it's written in C, doesn't need a VN to run? Or because its messages are stored in RAM not SSD?

erulabs3• 11 months ago

I haven’t used Kafka since the change to remove zookeeper, it’s likely they’re more or less on par now. Redis gets a win because most shops already have Redis, it’s already trusted and installed, just waiting for its first XADD.

enether7• 11 months ago

Love the story!

Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.

Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)

And why was it a big step - what other systems didn't embrace dropping the C?

erulabs3• 11 months ago

Well at the time (and this is still largely true), people were very insistent on ACID compliance in databases. Obviously this made sense of many applications, but became a bottleneck at huge scale. Being able to be eventually consistent became a golden feature. It was worked around by using eg read replicas in production, as SQL replication breaks the Correctness in favor of the Availability. Kafka’s “acks=1” is part of the story yes, but simply writing events to be eventually processed also accomplishes “dropping correctness”.

Native support for dropping correctness in SQL is tricky, see Transaction Isolation Levels, but I mostly mean in overall web architecture, rather than specifically in one database or the other.

jiggawatts• 11 months ago

that SQL databases are _not well suited_ to drinking from a firehose of writes

Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…

atombender7• 11 months ago

For a long time I've wondered if we could just invent an extension for Postgres that allow physically ordered, append-only tables.

The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.

Kafka is of course more than that, but I bet you can get quite far with this.

pluto_modadic• 11 months ago

lots of people not considering UNPHAT - https://gist.github.com/rponte/67c78e6b3ee349a6e14cb8fb155b7... / https://stevenschwenke.de/pragmaticSoftwareEngineeringUNPHAT - some solutions didn't exist in the past. you evaluate your landscape and your problem, and try to find a good fit.

chatmasta2• 11 months ago

Kafka was created at LinkedIn when it was already quite a large web platform, to solve the problem of distributing a unique newsfeed to a wide audience… so while they weren’t Google, they were still closer to Google than the targets of the “UNPHAT” expression. You can apply UNPHAT to most of the people using Kafka today, but it’s not fair to apply it to LinkedIn when they created it.

pss314• 11 months ago

LinkedIn recently announced that it transitioned from Kafka to Northguard. Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2]

[1] https://www.linkedin.com/blog/engineering/infrastructure/int...

[2] https://www.youtube.com/watch?v=RDV6-MUVEbQ

kafcausir• 11 months ago

I haven't been following kafka for some years now, but i thought Linkedin were heavily invested in it. What happened? Also what happened to Confluent? Their team were ex-Linkedin members from what i remember?

pm90• 11 months ago

Confluent is expensive and I don’t believe LI used them; they did use OSS kafka. Im guessing that after being acquired by MS they explored other tech.

gdbsjjdn• 11 months ago

NorthGuard looks like a clean sheet redesign of Kafka. The OSS Kafka community has taken a long time to implement things like KRaft, which addressed metadata scalability concerns by storing metadata in the brokers themselves (it used to be in a separate data store called ZooKeeper which was operationally complicated). NorthGuard also supports splitting and merging ranges of keys without repartitioning the entire existing dataset. The way records are assigned to partitions is a big problem in running Apache Kafka at scale because it requires predicting the key distribution and number of partitions ahead of time.

Confluent is 11 years old and IPOed several years ago. It was founded by 3 ex-LinkedIn people who originally designed Kafka. 2 of the 3 founders are still at the company. LinkedIn never used Confluent, Confluent was a company founded to sell an enterprise version of the open source project (and later a cloud version).

ecoffey• 11 months ago

Northguard doesn’t look like it’s been open sourced? I’d be curious to know how it compares to Apache Pulsar [0]. I feel like I see some similarities reading the LI blog post.

0: https://pulsar.apache.org/

alt2277• 11 months ago

17 PB/day

Are they really generating data at this scale?

I cant even imagine a system which creates and stores this much data this fast.

enether7• 11 months ago

Where does this 17 PB/day number come from? I didn't quote any numbers directly.

Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)

djoldman2• 11 months ago

In 2010, LinkedIn had 90 million members. Today, we serve over 1.2 billion members on LinkedIn. Unsurprisingly, this increase has created some challenges over the years, making it difficult to keep up with the rapid growth in the number, volume, and complexity of Kafka use cases. Supporting these use-cases meant running Kafka at a scale of over 32T records/day at 17 PB/day on 400K topics distributed across 10K+ machines within 150 clusters.

https://www.linkedin.com/blog/engineering/infrastructure/int...

alt2277• 11 months ago

Sorry I meant to reply to this post above, who linked the blog post which mentions the 17PB/S

Post: https://news.ycombinator.com/item?id=45002089

game_the0ry• 11 months ago

This might be hyperbolic, but I think Kafka (or at least the concept of event driven architecture for sharing data across many systems) is one of the most under-rated technologies. It's used at a lot of big corps but is never talked about.

alphazard• 11 months ago

Like modeling everything as graphs, it is often a trap. Just because a model is flexible enough to capture all of your use cases doesn't mean you should use it. In fact you should prefer less flexible more constrained models that are simpler.

Distributed ledgers like Bitcoin do store transitions as events, and that's because nodes need the transitions to valid the next state. So you might say that Bitcoin is a widely run piece of software, using an event driven architecture.

Not every system needs to have all of it's state transitions available for for efficient reading. And often times you can derive the state transition from the previous and next state if you really need them (Git does that). Even though Git can compute all of the state transitions for the system, it doesn't store events, it stores snapshots.

djoldman2• 11 months ago

So now LinkedIn has dropped Kafka and wrote their own called Northguard. More info here:

https://www.infoq.com/news/2025/06/linkedin-northguard-xinfr...

According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).

Wtf is LinkedIn doing that they create 17 PB/day?????

rehathia• 11 months ago

Apache Kafka was originally developed by LinkedIn engineers, primarily Jay Kreps, Neha Narkhede, and Jun Rao, around 2010.

It was later open-sourced in 2011 and became a top-level project under the Apache Software Foundation in 2012.

The creators went on to co-found Confluent, a company that provides commercial support and enterprise features around Kafka.

dbacar• 11 months ago

The biggest strength of Kafka in my opinion is consumer groups.I have been using it since 2016 in at least 3 projects and it never failed, not that big workloads though (~100 messages/sec max). However it is a bit difficult to monitor and manage using only the out of the box applications.

nicoritschel• 11 months ago

I would take a hard look at Kinesis/Pubsub for these data volumes; should cost in the tens of dollars monthly.

macmar2• 11 months ago

In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational challenges.

Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:

Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.

Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.

Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.

The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.

From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.

Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.

supriyo-biswas• 11 months ago

This appears to be a LLM generated comment; if my assumption is correct - please do not do this here. Thank you.

macmar2• 11 months ago

No, man, this wasn't done by an LLM. I actually do Kafka implementations and follow exactly the same script I described.

Perhaps the fact that I'm not a native English speaker may have caused this confusion. I just made sure my text was written correctly in English with a translator.

Why was Apache Kafka created?

Related Stories