Apache Kafka 0.8.0 released

http://kafka.apache.org/downloads.html

by mumrah • 13 years ago

131 51 13 years ago

kafka.apache.org

mumrah7• 13 years ago

This is the first release as an Apache top-level project and represents many months of hard work.

A few of the major improvements (from https://archive.apache.org/dist/kafka/0.8.0/RELEASE_NOTES.ht...):

  * Intra-cluster replication support
  * Support multiple data directories
  * Many new internal metrics
  * Time based log segment rollout

Plus many bug fixes and other improvements.

hans0l074• 13 years ago

May I ask how Kafka compares with an AMQP solution such as RabbitMQ? Thank you.

ddorian432• 13 years ago

It keeps all messages in a log(sorted) and each client has a position of where they are in the log. So they can restart some parts, or skip some. This makes it very fast.

And it has sharding, which no-other messagequeue has (i think).

old_sound• 13 years ago

RabbitMQ supports sharding via exchanges

mumrah7• 13 years ago

Quora discussion: http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...

RabbitMQ developer on the kafka-users list: http://mail-archives.apache.org/mod_mbox/kafka-users/201306....

SO discussion on several queuing systems: http://stackoverflow.com/questions/731233/activemq-or-rabbit...

morkbot• 13 years ago

Quora link where you don't have to register/log-in: http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...

nine_k2• 13 years ago

I like the naming trend and expect releases of Apache Ionesco, Apache Lovecraft, and Apache Poe real soon now. </obligatory-joke>

nullymcnull• 13 years ago

Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?

RyanZAG• 13 years ago

Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.

ecoffey• 13 years ago

In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream.

EDIT: should add that morkbot had a great link too:

https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...

jbooth2• 13 years ago

It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC".

Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages.

So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).

xorgar831• 13 years ago

How does that compare to RabbitMQ's disk backed store?

recuter• 13 years ago

How does it compare to services like Amazon SNS, IronIO, 'etc in terms of costs and other tradeoffs?

idunno2462• 13 years ago

It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.

boredandroid3• 13 years ago

This is not correct: http://kafka.apache.org/documentation.html#design

:-)

mumrah7• 13 years ago

Use cases (from the project page): http://kafka.apache.org/documentation.html#uses

xal• 13 years ago

Kafka is an integral part of Shopify's infrastructure. It's brilliant but under appreciated technology.

A full company, scalable event bus like this can totally revolutionize the way you build services.

nemothekid2• 13 years ago

And thank you for the amazing go client.

capkutay• 13 years ago

"A full company, scalable event bus like this can totally revolutionize the way you build services."

Shameless (and shameful) plug, but if anyone wants to be part of such an enterprise that's already gained traction with big companies, send me a message!

pspeter3• 13 years ago

Congratulations to the Kafka team! Their work is always extremely impressive.

x3942• 13 years ago

Finally!

harichinnan• 13 years ago

Was this developed in Scala?

boredandroid3• 13 years ago

Yup.

mumrah7• 13 years ago

It is a mixture of Java and Scala, though mostly Scala. GitHub mirror of the source: https://github.com/apache/kafka

shmerl6• 13 years ago

Why is it written in Java? Isn't it a default performance hit? I'd expect such frameworks to be written in high performance languages.

nine_k2• 13 years ago

Java is pretty high performance, usually on par with C++, and usually faster than e.g. Go.

Writing Java code so that there are no perceivable GC pauses is an art, but it is not impossible to achieve.

JVM might require more RAM upfront, but a well-written program is usually reasonably memory-efficient, too, so the consumed memory grows reasonably slowly with the problem size.

Writing things in pure C is often just too time-consuming.

shmerl6• 13 years ago

Java is pretty high performance, usually on par with C++

I'm not convinced. Java I/O is far form perfect, and Kafka is probably very heavy on I/O side.

and usually faster than e.g. Go.

That's strange, since Go to some degree was intended as replacement for Java without having Java's downsides. Why would Go be less performant?

I'd be interested if someone would write such framework in Rust though. C++ is of course a default expectation, but usage of Java somehow surprises me in this case.

BertrandP2• 13 years ago

Definitely, definitely not true. Go is extremely fast, it has its strengths in some areas and trade offs in other areas. If you want a good comparison to Apache Kafka, you should look at Apcera's NATS server, a publish-subscribe and distributed queueing messaging system written in Go by Derek Collison: https://github.com/apcera/gnatsd

You should never discredit a language, especially with blanket terms such as "its faster than Go". In what respects and in what areas? Here's a blog post which performs benchmarks on Go and Scala: http://eng.42go.com/scala-vs-go-tcp-benchmark/

They found Go to perform better than scala, however it had a high footprint. Every language has its tradeoffs, Java and Go are no exception.

mumrah7• 13 years ago

Java I/O is plenty fast. Just take a look at Hadoop, Lucene, Jetty, Netty, or any number of other pieces of high performance software written in Java.

Here's a good writeup talking about sequential I/O in Java: http://mechanical-sympathy.blogspot.com/2011/12/java-sequent...

shmerl6• 13 years ago

Thanks for the pointer. Is there any comparison with C/C++ I/O in various operating systems?

Note, that unlike Hadoop, original Google's map reduce system was written in C++.

turingbook• 13 years ago

I wrote about this release and Kafka in general in Chinese and gather some information perhaps useful for Chinese guys: http://geek.csdn.net/news/detail/3866

rmk2• 13 years ago

This has nothing to do with the software itself, but it bothers me: Why do you call a "high-throughput distributed messaging system" Kafka? Kafka's stories essentially describe the polar-opposite: crippled, ineffective, labyrinthine message systems that are exceedingly hierarchical in nature. They are also rather user-unfriendly, i.e. their users usually die horrible, lonely deaths. Am I just missing the in-joke here, and it's called Kafka because it is exactly the opposite of this, or did somebody overdo the hipster naming scheme?

mst2• 13 years ago

It's written in Scala, so perhaps the reference is to the source code rather than the client interface ...

... or maybe you're just overthinking it.

cleaver• 13 years ago

That was my first thought. Based on the novels I've read, I would interpret it in a couple of possible ways:

- You receive a message, but the system can't tell you why your received it nor what you should do. (The Trial)

- It's not a distributed messaging system with bugs. Actually, you are the bug. (Metamorphosis)

As an aside, I went to a tech conference in Prague two months ago and visited Café Slavia, a hangout not just of Kafka, but also author Milan Kundera and president/poet Václav Havel. I had a glass of absinthe in their honour.

boredandroid3• 13 years ago

The rationale was that it should be a writer because we were building a distributed log or journal (a service dedicated to writing). The writer needed to be someone that (1) I liked, (2) sounded cool as a name, (3) was dead (because it would be creepy to use the name of someone who was alive).

mumrah7• 13 years ago

At LinkedIn, they started code naming projects after authors and literary works. E.g., Kafka, Camus, and a bunch of Harry Potter references. Kafka was donated to Apache by LinkedIn and the name stuck.

Apache Kafka 0.8.0 released

Related Stories