I reckon your message broker might be a bad idea

http://programmingisterrible.com/post/81015328859/i-reckon-your-message-broker-might-be-a-bad-idea

by tonyg • 12 years ago

63 32 12 years ago

programmingisterrible.com

grumblestumble• 12 years ago

TL;DR - "bad implementations are bad"

plainOldText• 12 years ago

If you really want to tl;dr it, I would use this last paragraph:

"Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour."

Spearchucker• 12 years ago

It's why, in this agile world, I prefer noobs to use cases to user stories. Most templates I found explicitly prompt you to think about fail conditions. If you're wondering what constitutes being a noob, it's when someone is too inexperienced to think about fail conditions and work them through without being prompted by someone or something, like your app crashing out.

Note that I have nothing against inexperience - I used to be too. And noobs are malleable - in that sense they're easier to work with than experienced people.

cmbaus• 12 years ago

What the author is alluding to is that centralized messages queues can cause catastrophic failure over[1].

Industry knowledge of how to distribute loads has improved significantly in the past 10 years. I wish the author would touch on some other options like using a load balancer with consistent hashing[2].

[1] http://martinfowler.com/bliki/CatastrophicFailover.html [2] https://en.wikipedia.org/wiki/Consistent_hashing

glenjamin2• 12 years ago

Based on my experience with RabbitMQ, failed messages (ie. unacked due to consumer crash) have a retry count property that is maintained by the broker.

If the consumer simply checked if the retry count was above a threshold, and then redirected the message to a dead-letter-exchange, this would solve the "stuck message" problem.

As for tracking build up of messages, this is just a case of adding something like a nagios check on size of queue.

It seems like the solution outlined in the post is just implementing by hand the features of a decent message broker they chose not to use.

Sounds like they managed to get the job done regardless.

twic3• 12 years ago

I don't think RabbitMQ has a retry count. It has a retry flag, which tells you if this is the first time a message has gone out for delivery, or an attempt at a redelivery. You can use that to requeue after the first failure, but to reject (and send to a dead-letter queue) after the second. But you can't do anything more than that without manually managing a count.

At least, that's the conclusion i came to when i looked at redelivery for an application i work on that uses RabbitMQ. Please do correct me if i'm wrong.

glenjamin2• 12 years ago

You are correct, I was mistaken. https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.del...

The spec has a SHOULD for aborting after a retry count, but it doesn't appear to be exposed to clients.

georgemcbay• 12 years ago

I don't like sniping people about things that are unrelated to the actual point they are trying to get across; but in this case the problem caused me to just quit reading the post after the first paragraph, so....

I reckon the blog author should really reconsider the black on dark-grey color scheme of the blog; contrast is not your enemy when it comes to a simple, text-focused blog. This might look pretty decent on whatever system it is that he is writing it on, but on my PC monitor (which is quite a good one, carefully color calibrated and brightness reduced for editing photos), it is a complete disaster.

userbinator2• 12 years ago

View -> CSS -> Off

(What, your browser doesn't let you turn off stylesheets? :-)

brassybadger• 12 years ago

This is the old debate whether applications should be tightly or loosely coupled. As the article concludes, there are tradeoffs to consider - the broker is a SPOF indeed, but on the other hand it'll take care of all the queueing logic (which is not that simple as it seems - acknowledgements, backpressure, redelivery, timeouts etc etc). It's also much easier to scale the system out if applications are loosely coupled.

Furthermore, a good broker supports some kind of failover. We run a RabbitMQ cluster behind a load balancer, and it works great - once e.g. we had a hardware failure, and the applications did not notice anything from it. Of course we also have to monitor the queues on the broker (one of the few ways to bring down Rabbit is to fill up its disk). But having a central broker for connecting apps also means you have a central point you can keep an eye on to check where your bottlenecks are (which queues build up).

Loosely coupled applications are also much more unix-like in the sense that you can make them do one thing instead of baking in your custom restart logic (over a network? please don't).

HeyImAlex• 12 years ago

I feel like it's an endless battle, trying to move away from a SPOF; even in your rabbitMQ cluster your load balancer is a SPOF. Who watches the watchmen? At a certain point you just have to accept that you're not gaining a whole lot by adding more complexity. Process supervision over a network (a la something like fleet or flynn or mesos) is a pretty neat solution. I've been playing around with this stuff the past week and I'm having a lot of trouble coming up with the perfect setup haha

vidarh5• 12 years ago

A load balancer need (should) not be SPOF: You run two (or more) of them, with IP takeover (e.g. via keepalived or similar, that continuously vote on the master), or you make the clients try to connect to two or more IP's. Or both.

It's really quite trivial to ensure your load balancer is not SPOF if you're ok with clients having to reconnect on failure, and only slightly more hassle if you want to allow connections to survive one of your load balancer boxes failing.

twic3• 12 years ago

There's a whole other axis of pain around message queues that isn't mentioned, and that's regulating delivery. If you're pushing messages to workers using HTTP or whatever, then you can regulate delivery at the sender, or at a load balancer in between. If you're pulling messages via an MQ protocol, you can only do it at the receiver. I'm not aware of any message broker that lets you throttle messages on a per-consumer basis, much less one that does this automatically based on load or health checks.

Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself.

Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick.

Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem.

Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load.

You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work. It seems to be considerably more involved than doing it with HTTP. Basically, using a message queue pushes a concern which should be external to the worker into it, and that creates work that needs to be done.

I am coming to think that whilst it's okay for apps to send messages to a queue, they should never receive consume messages from one. Rather, i suspect there should be some small infrastructural component which pulls messages off a queue and sends them to workers with HTTP, via a load balancer. I think this component can follow some simple, generic rules, and leave the smarts in the load balancer as normal. I haven't fully fleshed this idea out yet, though.

jallmann• 12 years ago

It sounds the real problem is there might be a lack of production caliber load balancers for $MQ, rather than HTTP possessing any intrinsic advantage.

How is load balancing anything but an implementation detail, since one can usually be written as a service utilizing the same protocol in question? That's how HTTP reverse proxies usually work (notwithstanding other methods of balancing load such as DNS).

Other monitoring aspects such as feedback are already built into well-designed MQs. If that feedback isn't being acted upon properly, then that's again a problem with the implementation -- the same could be said for a HTTP-based task queue that simply ignores errors. It's not "extra work"; it's just different.

comice• 12 years ago

"I'm not aware of any message broker that lets you throttle messages on a per-consumer basis"

depending quite what you mean here, ActiveMQ supports this.

"With an MQ, no component of the system has a global overview of load."

except the MQ surely? Again, ActiveMQ supports multiple consumers on one queue and handles balancing the messages to workers at the rate they require.

vishnugupta• 12 years ago

Rather, i suspect there should be some small infrastructural component which pulls messages off a queue and sends them to workers with HTTP, via a load balancer

AWS SNS can deliver a notification to an HTTP/HTTPS end point. If your design permits the use of a hosted notification service then I highly recommend you explore SNS.

And yes, I do agree with all the concerns you have raised! In my company we use Kestrel with a couple of consumers. Load asymmetry and worker health check is a real pain for us.

vidarh5• 12 years ago

If you're pushing messages to a message broker, you can trivially regulate delivery on most of them by checking queue depth and throwing errors or reducing the delivery rate if they grow beyond a certain threshold.

But if you genuinely have more messages than the workers can process, pretending you don't is not solving a problem. With the broker you have the choice to decide what size of queue is acceptable without building your own queue logic into the client. The only case where HTTP is generally preferable there is the case where a queue of any size is always unacceptable: If you need the message processed right now, or not at all.

Deploying a new worker instance? With HTTP, you don't add it to the load balancer pool until it's ready. With an MQ, you have to tell the app to add itself.

I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.

Worker failed a health check? With HTTP, you take it out of the load balancer pool. With an MQ, you have to tell it to take itself out of the pool, and hope it can still do that even though it's sick.

With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken - why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?

Worker failing to process messages? With HTTP, there's a good chance you'll notice straight away because it's returning 500 errors. With an MQ, you have to hope your monitoring will detect the problem.

What is the difference between monitoring for 500 errors and monitoring for errors from an MQ? Something needs to generate those errors. Either you are monitoring the logs, or your app is sending responses directly. If you need/can use synchronous responses, either you are abusing a message broker for something it shouldn't be, or your app should be sending reply messages that the client needs to wait for (which may still be preferable if the processes are potentially very long running). There's no difference here.

Workers loaded asymmetrically? With HTTP, the load balancer is in just the right place to, er, balance load (think of Apache mod_jk or mod_cluster). With an MQ, no component of the system has a global overview of load.

With workers pulling work out of a message broker, they self-regulate: The ones on the highest loaded machines will pull messages slower. Load is usually the wrong thing to worry about from the app perspective - processing rate is much more relevant, and a metric that's trivial to make available from your app. Latency is easily handled by imposing strict limits and killing / pausing workers that miss them.

You can make the workers do the right thing in these scenarios even when using an MQ, but it's extra work.

Frankly, in most of your scenarios I feel the HTTP alternative is what poses the extra work. You're imposing extra steps to manage adding workers to the load balancers; you're adding complicated load monitoring instead of letting the workers self regulate which ones have capacity to process the messages just by virtue of which ones pull the next message first.

Brokers are not always the right choice. They're generally the wrong choice if you want synchronous, low latency end to end processing. But for many systems, different parts of the system may process different types of jobs at different rates, and with different scale out requirements. In those cases, introducing brokers means each component only has to worry about their specific task, as the overall data flow management gets separated out in a separate component monitoring the brokers and taking appropriate action (such as, for example, spinning up more VMs to run brokers at some point in the pipeline).

twic3• 12 years ago

If you're pushing messages to a message broker, you can trivially regulate delivery on most of them by checking queue depth and throwing errors or reducing the delivery rate if they grow beyond a certain threshold.

I'm not sure what you mean. Are you talking about regulating messages going from clients to the broker? I was thinking about regulating messages from the broker to the workers. I haven't seen a broker which will let you regulate the flow on a per-consumer basis like that.

I much prefer having the app adding itself - having the load balancer control delivery means it needs to probe for liveness, and the config needs to be updated whenever you bring resources up/down. If the endpoint can register itself, then that's one less hassle.

It's one less hassle in the control plane, and one more in the worker. I would prefer to have this stuff in the control plane rather than the worker, since it's not something directly concerned with the work being done. Separation of concerns and all that.

With an MQ, it will stop pulling messages if it's not functioning. If it's functioning enough to keep pulling messages despite failing a health check, your app is broken

Yes. Exactly. I would like the system to do something sane in the face of broken workers.

why was it not doing sufficient checks itself to know that it is not working? Why is there not a monitor on the same physical server with the capability to kill the offending process?

That's quite a bit more work than just taking it out of the pool. And killing the server isn't necessarily the right answer - if the instance is locking up because it's going a full garbage collection, for instance, the best thing is probably to let it finish and then put it back in the pool.

What is the difference between monitoring for 500 errors and monitoring for errors from an MQ?

500 errors are thrown straight into the client's face. The client itself can log that something went wrong. The closest analogue with an MQ would be for the broker to log when a consumer rejects a message. Can you configure them to do that? I honestly don't know. You'll struggle to have the logs tell a story which makes as much sense, though, because the MQ doesn't have the context that exists in the sending app.

Don't get me wrong, i don't think message queues are worthless. There are problems for which they are indisputably the right solution. But i think people should be aware of the operational challenges in using them. I have seen a lot of naive enthusiasm for queues which doesn't seem to take this into account.

Gigablah• 12 years ago

We’d ignored and subsequently embraced three good design principles for reliable systems: Fail fast, Process Supervision, and the End-to-End principle.

Can't you have a message broker AND embrace those design principles... without pushing responsibility onto your crawler? What happened to the Single Responsibility Principle?

joelberman• 12 years ago

These reckons you mention. Over the years I have come across people suggesting the same approach to a problem. It makes sense in theory and it should work. Yet it doesn't. And I cannot explain why it does not work. Queueing/throttling/steering/pub-sub, caching, and synchronization seem to be three areas that spawn unintended consequences like crazy. If some young'un suggests something you have learned will fail, let them try it if you have the time. And then ask them to explain why it did not work. It's fun watching people learn.

fallous2• 12 years ago

"Used message queue as replacement for RPC service without actually implementing RPC pattern." Key learned from inevitable failure? Message queues are BAD! Except for the part where you weren't actually testing an MQ, you were trying to use an MQ as an RPC service without actually solving the problem and instead insisted on building an RPC service with none of the failure-tolerance of a persistent broker.

I don't disagree with the OP that this is a common mistake, but the solution offered is more of a "we also made this mistake, so clearly it is a problem with MQs and not us!" which isn't actually true.

Additionally, MQs are really (IMO) a better fit for systems that either have a measurable level of complexity on writes or are lightly written but heavily read. You see the same general pattern failure with people who insist that RDBMS is The Answer(tm) even though the data they write/read isn't relational and often read-heavy.

fallous2• 12 years ago

I want to make clear that I'm not intentionally faulting the OP in this situation, because the decision with regards to message brokers isn't easy in a complex system. Often you have to mix "lossy" messages (pubsub) with simple ack messages (receipt confirmation) as well as full RPC-style messaging (receipt ack, result response) and that is usually a non-trivial endeavor from both an architectural as well as implementation standpoint.

csense• 12 years ago

Every time I've looked at message brokers, I've been struck by how over-engineered they seem to be. For e.g. the distributed crawler example he gives, I'd rather write a database-backed RESTful service myself than use something like RabbitMQ, because with the former at least I have a nonzero chance of understanding what the hell it's doing and what the potential failure modes are.

Whereas every message queue system I've ever seen has incredibly complex documentation, and you have to study it for quite a while to even figure out what the message broker's model of the problem domain actually is.

I don't know enough about message brokers to say for certain they wouldn't be right for certain really complicated commercial sites with extremely high load, where you can afford a full operations team with one or two people who specialize in understanding message broker technology, but for anything smaller than that, to me it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology.

userbinator2• 12 years ago

They're just another one of the numerous examples of enterprise software out there. Overgeneralised and overabstracted, promising everything and probably capable of it, but not doing any one thing particularly well while consuming inordinate amounts of both machine and human resources... and they somewhat mirror the bureaucracy of the companies that use them.

Thankfully I don't work in that space, but just reading some of the descriptions of these systems and software (and "platforms", "solutions", etc.) is enough to make my head spin.

bananas2• 12 years ago

I do work in that space and the reason we use stuff like this is that we can't afford to do the processing up front as it reduces our capacity and makes people wait. The MQ solutions allow complicated logic to be deferred until later, allow it to take as much time as it needs (sometimes our transactions are open for hours or even days) and allow us to do integration and calls to other systems.

Enterprise software has turned into a somewhat derogatory term thanks to conjecture but the stuff you don't hear about works pretty well and performs like nothing you've ever seen before.

uniclaude• 12 years ago

it seems like a message broker is an over-engineered piece of garbage you shouldn't ever use in projects that demand reliable, well-understood technology

This is not because you do not understand how message broker works that they are not simple. Message brokers implement "reliable, well-understood" protocols, and you do not need to have engineers who "specialize in understanding message broker technology". This is downright FUD. The little complexity of those protocols is here for a reason, and, seriously, we're not talking Enterprise Java complexity.

It's just like saying that you could use bash scripts to automate your infrastructure unless you have Google scale, because Puppet, Chef, or Ansible are too complex and over-engineered.

Writing a database-backed RESTful service for a message queue is not without reminding me of those who reimplement RDBMS over MongoDB because RDBMSes are "over engineered".

If the AMQP protocol is too complex for you, there are other, simple protocols that message brokers implement.

vidarh5• 12 years ago

Message brokers can be very simple. My first production Ruby app was a message broker that took all of about 700 lines of code. An indeed I wrote it because I'd rather deal with something I knew intimately than deal with one of the over-engineered brokers.

But they are often complex because they offer a lot of complicated options for failure scenarios, handling of cases where a worker hangs (e.g. option for the broker to put the message back on a queue automatically unless the worker regularly proves liveness), different levels of resiliency (e.g. memory only queues, disk backed queues, or even replicated queues with distributed transactions to guarantee in-order, single-delivery even with a failed servers etc.). If you need that, then the investment in learning one of the complicated brokers may be worth it.

Otherwise, there are plenty of simple "do it yourself" message broker options if you know your specific requirements. E.g. LISTEN/NOTIFY in Postgres (or a table; add synchronous replication and scripts to automate failovers and you have a very resilient solution), Redis lists/sets, or even a mail server (I'm not joking - I've used Qmail as a queueing server in the past; for some things it works great)

vidarh5• 12 years ago

So here's an alternative view of how he could've achieved what he wanted:

- Use a broker that supports a timeout on the messages. Many do. E.g. beanstalkd lets you reserve a message for a specified time period, otherwise it goes back on the queue automatically. You can let the server knows if you need longer to finish. But a locked process will not hold on to a job.

- Use a timeout on the crawlers. A hard timeout like alarm() that causes the process to die if it fails. Run them under a process monitor that'll respawn them.

- Have the crawler or a separate system monitor queue depth, and slow down / alert/ throw errors if the queue goes above a certain depth.

All of this is trivial.

If introducing a broker takes the whole app with it if its fails, then you're doing it wrong: Unless you need in-order guaranteed delivery of messages, making a message queue resilient to failures is trivial: Just add more of them, and let the client talk to all of them. If you need in-order delivery, then a active-passive setup is easy enough too. Put the queue in a synchronously replicated database if you can't afford to lose any of them. So his argument about failures, is an argument against letting "inexperienced programmers" set up systems with single points of failure, not against message brokers.

The irony of this is that his solution is that it's great when the process generating the messages to queue can be run on a single, reliable system, by someone who will notice when that system fails and recover from it. But you've now stuffed your crawler full of state, and created a central point of failure. Just what he complained about.

The broker solution is great when you want scale - you can multiply the number of brokers. You can multiply the number of machines the workers run on. You can multiply the number of machines processing the crawled documents. It is also great when you want resilience for the same reason: By decoupling, it makes it easier.

If you don't need that scale, then go for the simpler solution. But if you do need that scale, then his solution isn't a solution at all.

If you introduce a broker, and your queues start growing out of control. Guess what? You have a scaling problem you would likely have without the broker too. Only in a decoupled system you have a much easier way out: Add more machines/VMs with workers; add more brokers.

Don't feel you have a way of controlling it? This is what your monitoring and orchestration systems should be doing. Queues over a certain threshold? Trigger an alert, and optionally spin up an exra VM.

angrybits• 12 years ago

"All of this is trivial."

No, describing it in English on HN is trivial. Getting all of this to work in the really real world is incredibly difficult and time-consuming. Components and infrastructure will inevitably find novel ways of failing that your three-bullet architecture didn't take into consideration.

That said, I agree that this is a problem that can be solved with a message broker. I just don't think it is easy. If you can make a system work with fewer moving parts, then you probably should.

I reckon your message broker might be a bad idea

Related Stories