The Big Little Guide to Message Queues (2020)

https://sudhir.io/the-big-little-guide-to-message-queues

by throwup238 • 2 years ago

185 44 2 years ago

sudhir.io

raju• 2 years ago

Previous discussion on HN with 108 comments—https://news.ycombinator.com/item?id=25591492

actionfromafar• 2 years ago

Whoa, the top comment is really +5 insightful.

corford• 2 years ago

Yeah, it's a great comment. Since then things like Temporal have sprung up to try and address this state machine complexity of "accidental decoupling"

Terr_2• 2 years ago

Accidental decoupling is where you have a complex state machine encapsulating a business procedure with multiple steps, and it's coordinated as messages between and actions in multiple services.

That might need emphasis on "in multiple services."

Within the same service, a granular set of messages (events) can still be useful for auditing or creating good read-model "projections" of what happened.

bvrmn3• 2 years ago

I could add 2c. If you ever need to store some meta alongside the message in a DB. For example status, some execution history, etc. Then it's better to avoid MQ at all. Of course if you can scale DB access from workers and you can couple producers/consumers via the same DB. But it's the case for almost all applications TBH.

kapilvt• 2 years ago

thanks for noting, 2020 should be in the title, I was about to comment that event bridge (which linked article notes is new) is a rebrand on cloud watch events which have been around since January 2016.

thezorg• 2 years ago

Poor [Cloud Tasks](https://cloud.google.com/tasks), they're the actual GCP message queue, but everyone forgets they exist and use Pub/Sub instead.

What's funny is Pub/Sub is a fundamentally different model from message queues: queues are normally tightly coupled, meaning you enqueue a message to be processed by a specific system. Pub/Sub on the other hand is fundamentally loosely coupled: the publisher doesn't care who (if anyone) gets the message.

(I've heard "orchestration" vs "choreography" to describe this dichotomy, but I can't say I'm a fan of the jargon.)

xyzzy_plugh• 2 years ago

Pub/Sub is truly the stuff of nightmares compared to Tasks. I wish Tasks were a little more loved, because I keep ending up using Pub/Sub inevitably.

candiddevmike• 2 years ago

Pub/Sub is meant to follow the Kafka "insane amounts of data" firehose pattern. Everyone thinks they need that scale unfortunately and skips over Cloud Tasks.

Spivak• 2 years ago

The Kafka pattern isn't about the size of your data, it's about accommodating heterogeneous consumers and easy fan-out. I don't love the ampq exchange model because it adds state to the middle of the queue.

sudhirj5• 2 years ago

Wasn’t this on AppEngine in the early days? Yeah, if this is the same thing it’s probably the first queue I learnt / used. Will add it in.

swyx2• 2 years ago

(as someone who has pitched orchestration vs choreography) why don’t you like the jargon?

magicalhippo2• 2 years ago

We've started using message queues to decouple legacy code from new code in order to migrate from one platform to another. I modelled our implementation after AMQP so we could trivially switch to either RabbitMQ, Azure Service Bus or similar down the line.

However one thing that seems missing from the standard is retry policies. Sure you could just abandon the message and rely on the timeout, but I'd prefer an exponential backoff in case some external service is down for an extended period to avoid things bogging down.

Are there some standard ways I've missed, or do folks rely on proprietary extensions or extra services for this?

dmoy• 2 years ago

Are there some standard ways I've missed, or do folks rely on proprietary extensions or extra services for this?

As a hack, you can always have your library run its own retry by doing an atomic ack-and-resend-to-the-future (though you need to have bits for retry count if you want exponential back off). And there's situations where it doesn't work well, if the message handler itself crashes too hard on failure.

magicalhippo2• 2 years ago

I mean yea I can do a lot myself given it's my implementation, but I was hoping to keep our messaging code fairly generic so it'd be easy to use either RabbitMQ or Service Bus, depending on if customer wanted on-prem or hosted installation for example.

fake-name• 2 years ago

It's not exponential backoff, but I've done this in RabbitMQ with some queue weirdness.

I have the main queue with a reasonably short timeout (30 seconds). That queue is set up with has a dead-letter queue where failed messages that don't get ACKed get moved to.

The dead-letter queue has a TTL of ~5 minutes, where it's dead letter queue is the original queue.

So basically, if a message fails a worker, it gets kicked over to the dead-letter queue, which then moves it back to the main queue after the TTL times out. This foes mean a crashing message will fail forever (so you have to keep a careful eye on how many messages are in the dead-letter queue), but I've managed to work around this so far. Or you can use proprietary extensions (x-delivery-attempts).

throwawaaarrgh4• 2 years ago

If you want to learn about Message Queues firsthand (and you should if you care about CS) I recommend implementing some with ZeroMQ. (I love the joke in the article about it being "zero effort" xD) There are many, many forms of queues, and you can implement most of them with ZeroMQ. There's tons of examples in many different languages, and lots of guides and docs to help you. Even considering all that, it will still be a pain to get the queues working, and that pain will help reinforce the purpose behind different queue architectures, and why it's so nice to have COTS message queues.

That said, if you want a customizeable FOSS message queue stack, use NATS. I'm not aware of a better, more complete solution designed for custom integration.

katzenversteher2• 2 years ago

I never used ZeroMQ (visited their webpage many many times though) but I totally agree that going through the pain, struggles and subtleties of message queuing is a very valuable and important learning experience.

I had those experiences with RabbitMQ around 10 years ago when I pushed the team I worked for towards a message queue solution to scale our distributed test system. It turned out successful but it took quite a while :)

sudhirj5• 2 years ago

Hey, author here, feels really nice to be HN famous (as defined by your stuff being on front page without you posting it). AMA and feedback welcome I guess.

Wanted to write a book on this topic over the pandemic lockdown, but then I felt it could just be a post. Let me know if anyone thinks it’s worth a full book. Maybe more exploration and demos of the options, reference designs etc.

akoboldfrying• 2 years ago

Really useful info at just the right level for me, and coincidentally at just the right time! Thanks!

One aspect I'm interested in is what happens when senders and receivers are mismatched in throughput, with the worst case being a too-fast producer causing queues to get "backed up". Do most of the tools you described have decent diagnostics/monitoring for this? Are there design-stage approaches to avoid this completely, even if demand varies (predictably or unpredictably)? Any light you could shed here, or in the post, would be appreciated.

swyx2• 2 years ago

this is known as backpressure in the biz. your queues eventually have to start returning errors as they get backed up, and those errors need to be reflected in the producers and ultimately fed up to the end user for feedback. no big deal

sudhirj5• 2 years ago

Depends on the queue... I've seen SQS queues holding millions of messages, and think I've heard of a billion plus somewhere - not sure other systems handle that kind of pressure.

In terms of monitoring, yeah, most systems will offer that... On AWS for example we setup cloudwatch alarms that post in our team chat / email when the number of messages waiting in certain queues reaches N, and we go see what's up. AWS also has autoscaling options with cloudwatch alarms, so some consumer services are configured to automatically add more capacity when there are more then N messages in the queue.

feverzsj• 2 years ago

Most users want to use a message queue in a transaction safe way, so they ended up implementing it in database.

mrkeen3• 2 years ago

But it's not enough - for a few reasons:

* Databases are not as transaction-safe (in their default setup) as one is led to believe. They are ACID for some non-isolated definition of I. It's popped up a number of times in the last few months on HN and was news to me: https://news.ycombinator.com/item?id=38736904

* Even if an ACID transaction protects you from concurrency issues, it won't protect you (more specifically - your data) from logic issues. Bad code will make your database data wrong, even after you push a fix to the code.

* The transactions can only span one database, which can limit your system design a bit.

Embrace eventual consistency and use a persistent, append-only message queue, and these problems go away. The downside is you'll be accused of chasing fads and doing resume-driven development ;)

valenterry3• 2 years ago

Databases are not as transaction-safe

Yes they are. Transaction-safe does not mean "protecting against concurrency issues". It just means that a transaction either succeeds or fails, nothing more and nothing less. Then there is isolation which is an orthogonal thing. So let's use the correct wording here.

it won't protect you (more specifically - your data) from logic issues

Nothing protects against logic issues though.

Embrace eventual consistency and use a persistent, append-only message queue, and these problems go away

I agree that eventual consistency should be embraced, but it doesn't make the mentioned problems go away. How would a persistent, append-only message queue allow for solve the C of CAP while retaining A and P? Of course it can't. How does it protect you from logic issues? Of course it can't.

another2another• 2 years ago

There are Transaction Coordinators that will allow you to span transactions across multiple systems - databases, OR MQ Brokers.

This means you can begin a DB transaction, enlist the MQ GET operation, and then commit or rollback the whole lot. It obviously slows things down a tad, but ensures consistency across all actors.

https://en.wikipedia.org/wiki/Microsoft_Distributed_Transact...

katzenversteher2• 2 years ago

On an intuitive level I love message queues. I like how they loosen the coupling between components and the amount of flexibility they provide. A few years ago I even built quite a large system production with RabbitMQ (a scheduling, monitoring and reporting system for distributed tests).

However, I find it extremely difficult to reason about them. What do I mean with that? I mean all the nitty gritty details like performance, bottlenecks, crashing participants, network issues, queues filling up, protocol issues that lead to message cascades and DDoS etc.

n4r9• 2 years ago

I hear you. I've owned a backend service using Azure Service Bus for several years, which recently ran into problems when a new deployment happened. It turns out they the default settings on the bus causes messages to be re-tried ten times if they take longer than 30s to process. That makes small issues escalate into big issues very very quickly.

sudhirj5• 2 years ago

Yeah, have learnt the hard way over many weekend midnights to know these numbers. This particular problem especially leads to exponential explosions - something needs to be pretty overloaded or computationally expensive to take over 30s (or queue cutoff), and retrying it adds more of the same load and makes other messages slower and therefore causes them to be retried again. Not fun.

justsomehnguy• 2 years ago

Ask HN: What message queue software has a native HTTP receiver?

RabbitMQ has one, but it's actually a management interface and I really don't want to expose it to the whole world.

X/Y problem: I need to send data from clients without custom protocols (so no libs bundled) over a plain HTTP/S (so it can traverse firewall in 99.9%) to the public endpoint.

I don't really care about the most features of the message queues, but I definitely don't want to write one.

I had a successful deployment on a very similar task with RabbitMQ, but it was fully internal and had no security requirements.

sudhirj5• 2 years ago

If you're up for AWS then the SNS-SQS combo works great. If you're not into giving short term credentials to clients then a lambda (with HTTP invocation) should work fine.

For bare servers maybe NATS? Think it had HTTP/S support.

masfoobar• 2 years ago

Started using message queues properly back in 2015 with MSMQ (Microsoft Message Queue)

Had an urgent change to make, away from sending data over from one company... to another. It was a big change for small amount of time. I decided to try MSMQ as it was already available on Windows PCs and it worked an absolute treat! Another feature was added a few months later, which created a lot of "messages" than the others but being MSMQ is handled it just fine!

Since then I have been interested in Message Queue technologies. Been a ZeroMQ (and NetMQ) guy since.

I ended up creating a light-weight broker with NetMQ (in C#) to handle my requests, sending data to Workers. It made in completely distributed, and can be placed on different machines, etc. I didn't need a dedicated server.

The broker was using push/pull or router/dealer patterns, and handled issues and loss well.

pub/sub was also supported but not as a "reliable" pattern. It was for notifications but can "retry" if not received.

I have dabbled in others like RabbitMQ and Kafka. Its just I don't need to use them. I guess the only exception will be starting a new job that has 'em. I have a light-weight Broker than I can install easily.

The Big Little Guide to Message Queues (2020)

Related Stories