Comparing Message Queue Architectures on AWS

http://tech.forter.com/comparing-message-queue-architectures-on-aws/

by itaifrenkel • 11 years ago

87 42 11 years ago

tech.forter.com

itaifrenkel18• 11 years ago

OP here... I would be happy to discuss any comments you have.

I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?

itaifrenkel18• 11 years ago

The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it was based on multiple processes communicating through files... which didn't scale (as one would expect). The Kinesis implementation was done by a researcher which is a very talented coder but did not have experience with Kinesis. He was up and running after a short whiteboard session and a few hours of coding. We had to tweak a few things in the following days (such as adding more shards for higher read throughput, abandoning Kinesis shards altogether in favor of multiple stream with one shard)

pan692• 11 years ago

For RabbitMQ you mention:

  No message delivery guarantee in face of RabbitMQ server failure.

Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?

itaifrenkel18• 11 years ago

Your are correct. The sentence you referenced is in the section that describes RabbitMQ without clustering. The next section talks about the tradeoffs of RabbitMQ clustering.

ekimekim• 11 years ago

Another very minor detail, but you mention that RabbitMQ can do priority based on multiple queues. While that's certainly a fine way to do it, it's worth noting that AMQP also supports per-message priorities within one queue:

http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basi...

The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinct levels.

Depending on your client, it may be difficult to prioritize consumption of one queue over another, so this solution could be preferred.

chwahoo• 11 years ago

Out of curiosity, what tool did you use to sketch the various architectures for your post?

itaifrenkel18• 11 years ago

I used Lucid Charts. It felt easy to work with.

https://www.lucidchart.com/ http://aws.amazon.com/architecture/icons/

halayli2• 11 years ago

Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other.

When you mention low latency, how low are we talking? ms, seconds, minutes? The reason I am asking is because you can use s3 as an intermediate storage where you ship your compressed logs/events at a rollover interval and the processing servers discover them there. Now of course this only works if latency is not a big deal.

itaifrenkel18• 11 years ago

Our first architecture was indeed peer-to-peer where all components discovered each other. It is difficult to isolate failures on this kind of architecture. A reliable queue makes a big difference in zooming in on the root cause of the problem (a queue producer or a queue consumer problem). Regarding latency - we are aiming for a few ms tops. preferably less. So bulk events using S3 was not considered

tylertreat2• 11 years ago

Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures."

FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments (https://github.com/tylertreat/Flotilla). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they compare.

itaifrenkel18• 11 years ago

We haven't done comparative throughput performance. SaaSiness, Low Latency and use case matching was the main concern of choosing the queues. As for throughput, the main concern was for the event stream processing pipeline. Redis and Kinesis are (somewhat) SaaSed on AWS, throughput can scale out horizontally with sharding, and we haven't had any problems since :)

itaifrenkel18• 11 years ago

Do you have a document showing the results of the Flotilla tests ... how beanstalkd compares with RabbitMQ for example?

tylertreat2• 11 years ago

Not yet. The blog post linked in the readme there provides some more background on the motivation behind the project.

I'm hoping to do some in-depth analysis of several brokers at some point, but I want to get the benchmark instrumentation right first.

yashinovsky2• 11 years ago

Have you also evaluated Kafka? as it is a common choice feeding Storm .

sargun• 11 years ago

Kafka is a log, not a queue. They have every so slightly different semantics, but logs have HOL blocking per partition for processing messages because they use a watermark, compared to queues, which track on a per-message basis.

rohitnair• 11 years ago

The article already includes a couple of "not queues" that they use for event stream processing, Kinesis being one of them. Kinesis consumers also maintain a watermark to keep track of how far they've read in a stream. Considering that, a Kafka comparison seems like a reasonable ask.

itaifrenkel18• 11 years ago

Sure we have. Kinesis was considered because it was inspired by Kafka, and Kafka is known to work well. The thing is that any distributed stateful service, by design, requires DevOps experience and takes time to manage. We try to stay lean at this stage, and chose Kinesis over Kafka becuase it took very little time to setup, and has no "maintenance costs". Had we have chosen Kafka, we would have enjoyed much shorter latency times.

eikenberry• 11 years ago

This is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment.

https://aphyr.com/posts/293-call-me-maybe-kafka

yashinovsky2• 11 years ago

For that matter, RabbbitMQ isn't either. see https://aphyr.com/posts/315-call-me-maybe-rabbitmq .

itaifrenkel18• 11 years ago

I always wondered if Amazon would really be open about and go public when a split brain in their network caused one of the hosted data services to loose data. Maybe they just throw money at the hardware and hope for the best.

threeseed• 11 years ago

Surprised they didn't mention IronMQ: http://www.iron.io/mq

It's not perfect but pretty damn close. And it run on AWS et al.

itaifrenkel18• 11 years ago

You are right about IronMQ, they are said to have a very strong product. For the record, we haven't actually checked redislabs and cloudamqp either, but since they are based on opensource that is mentioned in the blog post, I added them to the notes section. I couldn't say anything intelligent about IronMQ since it is not based on an opensource offering, and so other than actually trying it, I cannot tell its pros/cons.

jedberg2• 11 years ago

The first architecture using two ELBs has a long list of cons, most of which are solved by using HAProxy as your internal load balancer. May want to considering adding that as another option on the matrix.

itaifrenkel18• 11 years ago

Interesting. Could you please elaborate?

jedberg2• 11 years ago

Sure! Here's your list of cons and how Haproxy would solve it:

Some API requests need to get a higher priority over other API requests, and that is not taken care of. This was one of our main problems with this architecture, especially with a mix of real-time clients and clients that send batch jobs.

Haproxy would let you assign pools to different requests, each with their own priority and queue. At reddit, we had it broken down roughly into four quartiles based on 95th percentile response speeds of each API call.

This architecture assumes that there is enough Processing Servers to handle all requests (peak throughput). If there isn’t the Processing Server applies back pressure on the API server (error or timeout), which in turn returns an error to the API user which in turn re-tries the API request (applying more pressure). To avoid this, the number of running processing servers needs to be enough to handle peak traffic.

Using haproxy in the middle, that tier will queue the requests, so all the back pressure builds up at that second load balancer. Whether that is good or not is questionable, but at least you won't return errors to the clients right away. You'll still have to have a pretty long time out depending on how long it takes for resources to come online, but then you could go back to the first part and have longer or shorter timeouts based on the api call as is appropriate for your application.

ELB was not designed to handle huge traffic spikes since it takes a few minutes to internally scale. You should contact AWS support to warm your ELB if you have a planned traffic spike. We at Forter are in the eCommerce market where traffic spikes are rare.

This is still true, and there is no easy way around it unless you also make haproxy your front end load balancer. If you make it your frontend as well, you can have a hot spare on standby and spin up new ones pretty quickly. That being said, I believe the ELB team is making improvements in this area in 2015.

API Server needs to handle retries. Not provided by the ELB itself. Processing Server must respond within the http timeout (configurable between 1 to 3600 seconds). Otherwise the protocol would need two phases which adds more complexity.

Still true, although it will have to retry less because of the queues in the middle layer. Again though if you use haproxy as the front end you can solve this issue as well by sending the request to a "retry pool".

jpp123• 11 years ago

Out of curiosity did you look at gearman? I really like the ability to coalesce identical jobs.

itaifrenkel18• 11 years ago

Not yet. gearman is a job service which sits on top of a queue (much like python celery). We did not need all that, given the fact we are using Storm.

kondro3• 11 years ago

I'm surprised that SNS wasn't mentioned here, being an AWS queueing product.

itaifrenkel18• 11 years ago

Simple Queuing Service was mentioned (SQS) Simple Notification Service (SNS) was not. Did you mean SNS or SQS ?

kondro3• 11 years ago

Given that I was surprised that SNS wasn't mentioned, I think I mean SNS, given they are both queuing services.

omni2• 11 years ago

You give Redis a yellow rating on prioritization, noting that this can be partially achieved using multiple lists. Wouldn't it be much better to take advantage of Redis's sorted set type?

itaifrenkel18• 11 years ago

Sorted Set does not have properties of a queue. For example, it does not allow duplicates (different messages with the same priority)

omni2• 11 years ago

Stick a UUID on your queued items as part of the priority queue implementation.

Comparing Message Queue Architectures on AWS

Related Stories