Evaluating persistent, replicated message queues

http://www.warski.org/blog/2014/07/evaluating-persistent-replicated-message-queues/

by adamw1pl • 12 years ago

37 11 12 years ago

sargun• 12 years ago

What this article fails to talk about are primarily the failure scenarios of the queues under partitions. Does the queue become write-unavailable, read-unavailable, inconsistent, or otherwise during network partition? Can the queue remain partially available during network partition?

These are the situations, in a scaling, production system that really matter.

adamw1pl5• 12 years ago

These are very good questions, however for a different post. Here I was mainly interested in the throughput.

Jepsen post provide some insights (into Mongo and Rabbit). With Hornet you can easily end up with two primaries. SQS is untestable in that respect :).

baq• 12 years ago

required reading for anyone using distributed systems: http://aphyr.com/tags/Jepsen

in particular:

http://aphyr.com/posts/284-call-me-maybe-mongodb

http://aphyr.com/posts/315-call-me-maybe-rabbitmq

fasteo• 12 years ago

tl;dr Distributed systems don´t work as advertised; not even close

adamw1pl5• 12 years ago

Agreed, a must-read :). Bottom line is all queues can loose messages (we don't know much about SQS in fact).

whalesalad• 12 years ago

Would love to see Kafka represented here.

adamw1pl5• 12 years ago

Yes, my bad, should be there :)

Stay tuned, I'm out soon till end of next week, but will definitely add it.

srean• 12 years ago

I am not very familiar with messaging or message queues, so wanted to seek out some advice before spending time reinventing some well known wheel poorly.

I would like to have a broker that sits between an application and several endpoint services to which the application makes requests via the broker. These endpoint services have rules, typically throttling rules (i) you cannot submit more than x number of request per second (ii) you cannot queue up more than y number of requests at our end (iii) you must put a delay of z seconds after we have responded to a request of yours, etc., etc.

Is there a defacto standard open source software for this situation, that would take a config file describing the policies to follow for an end point. My requirement is pretty toy'ish, small to medium scale only. Just taking a look at Redis docs (am new to that too) it seems it will not be that hard to write out a simple solution. Would that be the way to go ? Or is there something out there that already handles this.

adamw1pl5• 12 years ago

HornetQ has flow control (http://docs.jboss.org/hornetq/2.4.0.Final/docs/user-manual/h...). Not sure about e.g. Rabbit.

However I'm not aware of a standard solution in that space.

Take a look at Reactive Streams (http://www.reactive-streams.org/) as well.

twic• 12 years ago

It would be nice to see the strongest of the various Apache contenders represented here too. Once upon a time, there was just ActiveMQ to worry about:

http://activemq.apache.org/

It's long been very popular, and has all the flexibility and enterprisey goodness you could want, but has not had a reputation for speed. However, i believe there was a release a year or two ago that significantly improved performance. I vaguely remember this being based on an infusion of technology from the Qpid project, which itself has a broker:

http://qpid.apache.org/components/java-broker/index.html

Or maybe that was what led to Apollo, which is the faster, more reliable successor to ActiveMQ, but which hasn't, er, succeeded ActiveMQ:

http://activemq.apache.org/apollo/

Or maybe i just imagined the link to Qpid.

I'm fairly hazy on how these bits of software relate, and i can't promise you there aren't more somewhere in the vast Apache mansion. Even figuring that out would be useful, never mind the benchmarks!

adamw1pl5• 12 years ago

I'm also not sure how the above projects relate. As far as the scope of the blog goes, Apollo seems to be ruled out as it is not replicated.

ActiveMQ does have replication, but they don't really sound convincing in the docs that it's first-class ;)

Evaluating persistent, replicated message queues

Related Stories