RabbitMQ 4.0

https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.1

by rhodin • 2 years ago

248 116 2 years ago

rhodin3• 2 years ago

This release includes a new (native, no longer a plugin) AMQP 1.0 implementation, new quorum queue features, and a new schema data store (Khepri)

sebazzz2• 2 years ago

AMQP 1.0 is great - then you can, behind the right abstraction layer, use it as drop-in replacement for Azure Service Bus or similar.

fidotron• 2 years ago

I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great.

Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.

leetharris• 2 years ago

This is how I feel about NATS: https://nats.io/

It's an infinitely more friendly version of Kafka, pub/sub, etc that is extremely lightweight. Yet every environment trends towards Kafka because it was "chosen" by the big corps.

packetlost3• 2 years ago

I've used both and NATS is definitely what I'd pick starting out. RabbitMQ is great too, but its heavier and harder to configure. Federation is the killer feature of RabbitMQ that NATS doesn't really have (afaik?).

shandor2• 2 years ago

So how about NATS compared to RabbitMQ? If building from scratch, what would drive a design or team towards NATS?

varispeed• 2 years ago

Big corps choose these bloated and complex tools because this way they can justify charging their customers premium and also that their customers won't be able to service such software on their own.

It's nothing to do with these apps being superior in any way - often it is the opposite - like reasonable faults and glitches can add more hours to bill the customers for "fixing".

frompdx• 2 years ago

I agree. NATS is much more simplistic to use and deploy. Easy to run locally for development. Jetstream offers useful features like persistent streams, and kv and object stores.

hendler• 2 years ago

https://nsq.io/ is also very reliable, stable, lightweight, and easy to use.

kragen• 2 years ago

nats looks more like a more friendly version of rabbitmq or zeromq than of kafka from these docs: https://docs.nats.io/reference/reference-protocols/nats-prot...

raverbashing• 2 years ago

Maybe because in typical developer fashion their website is absolutely dog-crap at explaining what it does, instead only throwing words here and there

"Key value", "pub-sub", "microservices can live anywhere" And?

Why do I care? What does "microservices can live anywhere" even mean and what does it have to do with NATS?

I find RabbitMQ sometimes inscrutable but I think even their website is better

nesarkvechnep• 2 years ago

Erlang is a secret weapon.

Joel_Mckay3• 2 years ago

RabbiMQ also graciously maintains a very nice Erlang repository for Debian.

Reminds me, I'll have to check if they have a working donation link someplace. =3

Alupis3• 2 years ago

Probably better said as "BEAM is a secret weapon".

BEAM languages (including Elixir and Gleam) share the benefits Erlang enjoys by also being part of the ecosystem.

Joel_Mckay3• 2 years ago

Probably using it wrong if complaining about AMQP queue scale limits...

Perhaps people are still thinking in single point of ingress design paradigms. Admittedly RabbitMQ can be a pain to administer for Juniors, but then again so are the other options. =3

smetj• 2 years ago

Agreed. I ran a log ingestion infra 8 years ago doing 20k msg/s sustained on RabbitMQ ... back then we went through a lot of instabilities though they settled over time with new releases. Great times. Besides a quality product the development/release process was very professional and mature.

The biggest issue back then was finding a quality client implementation for the language you were using. Not sure what the status of that is these days.

Joel_Mckay3• 2 years ago

AMQP and MQTT are both industry standard protocols. Also, RabbitMQ allows you to abuse the limits set by these standards.

Its unfortunate your team ran into performance issues, as Erlang can be inefficient in some situations. Were you using static routes on the DNS load balanced parallel consumers, or relying on the disk caching mechanisms?

datavirtue• 2 years ago

The real reason people don't use it is because they don't know about it or understand it. Then they apply the "it doesn't scale" retroactively.

You have to read a lot of docs or you WILL hold RabbitMQ wrong.

cogman10• 2 years ago

I agree, but there are a lot of footguns with RMQ. A great example of one is that you'll slow down your cluster by adding more RMQ servers (something that's bit us in the past). Which is a forgivable mistake as most people would expect that more cores == faster RMQ. (For RMQ, that doesn't work because Durable messages need to be replicated to the other nodes in the cluster. More nodes == more replication)

The ideal RMQ cluster has 3 servers and is dedicated to just a few apps.

latenightcoding• 2 years ago

It isn't more popular because it's not easy to use it properly.

I haven't touched it in years so I can't expand, but when I did, I had to write so many wrappers and add extra logic to use it properly.

gp• 2 years ago

I have found it to be more trouble than it is worth, as well

heipei2• 2 years ago

I could say the same thing about NSQ which is a distributed message queue with very simple semantics and a great HTTP API for message publishing and control actions. What it doesn't offer natively is HA though.

wejick• 2 years ago

People will be surprised on how far you can get NSQ. It doesn't come with any fancy guarantee like only-once or even ordered, this forced developer to think how to design better on the application side. Not saying it's ideal tho.

stackskipton2• 2 years ago

Also want to give a shoutout to BeanStalkd: https://github.com/beanstalkd/beanstalkd

If you are looking at RabbitMQ with "Maybe this is too much". Beanstalkd likely has features you need with almost none of the setup. Just don't web expose it ;)

anachronox• 2 years ago

Be careful with it, it will segfault randomly and there hasn't been a fix. After hitting my own posts on the Google groups while sleepily debugging the segfault at wee hours of the morning and getting falsely excited about the possibility of a fix, I gave up and wrote a replacement: https://github.com/chronomq/chronomq Have been running it for years without it falling over, submillisecond operations on average, and has processed billions of messages without failing.

petepete• 2 years ago

It's supported natively by Rails ActiveJob too!

I only used it on one project years ago and it was a pleasure, dead easy to get up and running and rock solid.

rjh29• 2 years ago

It's the polar opposite of RabbitMQ. It's a single binary written in C, you start it, you send text messages via TCP so writing a client or tooling is dead simple.

sevenf0ur• 2 years ago

How do you manage schema for exchanges, queues, and routing? The pattern seems to be for each client to set up the schema for the parts they touch before hand, but that doesn't scale well. The schema ends up siloed in each project and nobody knows what the current state should be.

ansd2• 2 years ago

Clients creating server topologies does scale very well with the new Khepri metadata store in RabbitMQ 4.0. Applications having an intimate relationship with their middleware (RabbitMQ) and creating flexible routing topologies is one of main strengths of RabbitMQ! RabbitMQ also allows operators to import (exchange, queue, and binding) definitions on boot. Server topologies can nowadays even be declared via YAML files in Kubernetes (https://github.com/rabbitmq/messaging-topology-operator). This way all the desired state is in one single place and the Kubernetes operator reconciles such that this declaratively declared desired schema is created within RabbitMQ.

brightball• 2 years ago

Is there a big advantage over Redis? Just coming from the Ruby world I know Sidekiq is extremely popular as a Redis backed queue. I know there's a RabbitMQ backed queue called Sneakers that's gained a lot of popularity as well though.

Just wondering what the biggest selling points are for somebody making a decision?

abe94• 2 years ago

We switched from a redis backed queue (bullmq) to rabbit at our company (https://github.com/laudspeaker/laudspeaker) when we hit speed issues . We found bull had a much lower throughput, after a lot of tuning compared even to rabbit out of the box.

We needed features like job priorities, and complex routing logic and on top of that our software needs to send millions of messages a minute. What we miss is a nice UI to see the state of jobs, and monitor for problems more easily like: https://taskforce.sh/

cloverich• 2 years ago

Been a while since I last used, but IIRC Rabbit is much more featureful than Redis, and has built-in much of what you'd get from Redis + Sidekiq. Their concepts[1] docs provides a good overview. This may be too broad a stroke but what I liked best about it was we used it for our centralized pub sub system, and although we had services in multiple languages / frameworks, they connected to each other through Rabbit, which saved us from having to setup a bunch of different / incompatible job processing systems. Like Redis its battle tested so you know what you are getting complexity / scale wise. At my current gig we use the typical Rails / Sidekiq setup and though it works fine I definitely find myself missing Rabbit. (But in my head Redis / Rabbit have only some overlap, and seeing both at the same company would seem totally normal)

[1] https://www.rabbitmq.com/tutorials/amqp-concepts

tejinderss• 2 years ago

At my current place, using Redis with celery is becoming bottleneck for number of concurrent connections it can hold. We are using 1000 workers and start seeing issues (ceiling is 10k connections in Redis); apparently Celery creates huge number of connections. We are considering moving to RabbitMQ for same reason.

bankcust083852• 2 years ago

Redis is slow and doesn't scale out well. Couchbase used to be a decent choice but it went commercial pseudo-FOSS.

ansd2• 2 years ago

Modern RabbitMQ scales well. RabbitMQ Streams provide a throughput of millions of messages per second. With native support for MQTT, more than a million users can connect to RabbitMQ: https://www.rabbitmq.com/blog/2023/03/21/native-mqtt

The new Khepri metadata store in RabbitMQ 4.0 allows for even better scalability.

rodrigobellusci• 2 years ago

Do you know of any book or video to get started with Rabbit?

Earlier this year I tried setting it up as a websockets message broker in a Spring Boot app but failed miserably. I ended up using Spring’s simple broker.

ofrzeta• 2 years ago

RabbitMQ in Depth https://www.manning.com/books/rabbitmq-in-depth The author is hanging out here, too.

joshlemer• 2 years ago

Do you have any recommended resources to learn how to apply these tools (RabbitMQ, Nats, etc) to typical enterprise services? Common patterns/usecases/best practices and things?

hztar• 2 years ago

whut.. transferred TB of data per hour with Rabbit.. does it not scale anymore?

hk1337• 2 years ago

I know this is said a lot about things people don't like or think doesn't scale but I think I a lot of people don't set it up and use properly and it doesn't scale doing their half-baked implementation.

matrix2003• 2 years ago

If you like RabbitMQ, check out NATS!

I can’t speak to the new version, but it comes with support for even more messaging patterns out of the box.

bankcust083852• 2 years ago

The anti-pattern to be avoided is cobbling together a nonperformant grand centralized ESB and making it a SPoF and bottleneck for everything, but it depends entirely on the use-case. MQTT scales to millions of devices for low data rates. ZK works well for little bits of coherent distributed cluster metadata. Kafka has its niches. ZMQ helps in others.

saberience• 2 years ago

I've seen it used for a company with way, way in excess of a million users. We used it for a system with 100M+ users for our login systems and in general all of our main account systems relied on it. Most of the brokers were always running at 15k to 25k messages per second.

I loved it and the only issues we had were due to our fuckups.

jimbokun• 2 years ago

For me the killer feature of Kafka is that topics are persistent until the data expires. Meaning different readers can be working at different offsets. And you can rewind or fast forward the offset you are reading, which can really be a life saver when things go sideways.

Does RabbitMQ have equivalent features?

rhodin3• 2 years ago

Yes, a stream queue type [0] is available where you can set retention, and replay messages.

[0] https://www.rabbitmq.com/stream.html

8n4vidtmkvmk• 2 years ago

I think people don't use it more because people don't really know what it is. From their website:

RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine.

What does that mean? "Messaging and streaming broker"? I understand the need for worker queues to process videos, images, emails and such but I can't easily tell if that's what this is.

Also, what are the benefits of this over just processing incomplete records straight out of my database? i.e. using MySQL as a queue.

halfcat2• 2 years ago

what are the benefits of this over just processing incomplete records straight out of my database? i.e. using MySQL as a queue

Mainly throughput and latency. I haven’t used MySQL recently so some of this may apply more to Postgres.

Postgres has LISTEN/NOTIFY which helps with latency. I don’t think MySQL has LISTEN/NOTIFY, which means you’d have to resort to polling.

You have to use the `SELECT … FOR UPDATE SKIP LOCKED LIMIT 1` features to grab a message from the queue table, so multiple consumers don’t pull the same message.

The biggest issue, if you’re trying to achieve decent throughput, is dealing with bloat (empty pages still sitting on the disk that haven’t been cleaned up yet). You can run vacuum but an online vacuum will only mark the pages as available for reuse (doesn’t free up the disk space). And if you run a full vacuum (which will free the disk space) it locks the entire database while it runs the vacuum. This can compound if you’re using indexes.

One way of dealing with this is setting up partitioning by message timestamp, so that as old messages roll out, you can just drop those partitions and not have to deal with vacuum.

It can work if your queue needs are low throughput or can tolerate higher latency, but there are some things to manage, and realistically setting up a Redis instance is probably less complex than trying to work around the database-specific quirks, unless you’re already very familiar with your database’s inner workings.

dyeje• 2 years ago

If you’re not worried about scale, I’d just use a database backed queue.

edweis• 2 years ago

For what reason should we move from SNS/SQS to RabbitMQ? Our SaaS processes ~20 events/second.

declan_roberts2• 2 years ago

That's like $0.50/day in aws costs. Absolutely no reason to switch if it's working.

RadiozRadioz3• 2 years ago

When you said that number, I had a completely different reaction. 20 messages per second is absolutely nothing, $0.50 per day for that is dreadful. A $5 per month ($0.16 per day) VPS can deliver many thousands of messages per second.

stackskipton2• 2 years ago

You TRULY NEED to be multicloud? That would be only reason. Otherwise, assuming SNS/SQS is meeting your needs well, I wouldn't even consider it. You are paying ~20 USD per month with no maintenance costs. That's hell of a deal.

pantsforbirds• 2 years ago

I probably wouldn't swap unless you have a specific complaint with SNS/SQS. I do like how easy it is to spin up Rabbit locally via docker-compose for testing, but I don't think that convenience is worth refactoring a significant portion of your code base.

Jenk• 2 years ago

I lost a lot of respect for the RabbitMQ maintainers when they refused to honor the semantic versioning scheme in package managers like nuget/maven/etc. "Safely upgrading" was impossible. 3.5 => 3.6 saw the removal of an argument.

They didn't lose my respect for the removal of the argument, however, they lost my respect for whatabouting the conversation calling SemVer a "no true scotsman" fallacy, then trying to claim that removing a redundant argument is not a breaking change, and other reality-warping nonsense, before blocking myself and other complainants from their issues - and even deleting some of their own comments to mop up some of their own terrible reasoning.

I'm sure there is no love lost on their side, either. Personal rant over.

57546gwg• 2 years ago

Based. fuck semver.

sebazzz2• 2 years ago

I lost a lot of respect for the RabbitMQ maintainers when they refused to honor the semantic versioning scheme in package managers like nuget/maven/etc. "Safely upgrading" was impossible. 3.5 => 3.6 saw the removal of an argument.

Well, at least AMQP 1.0 is now supported so I expect that for most things you are able to use any client now.

383toast• 2 years ago

does celery work with this out of the box?

jihadjihad• 2 years ago

It should--RabbitMQ is the default broker

https://docs.celeryq.dev/en/stable/getting-started/backends-...

phishhook2• 2 years ago

There are some limitations when using Quorum Queues, this was only patched recently - https://github.com/celery/celery/issues/9149

raminf• 2 years ago

RabbitMQ can also act as a native MQTT broker. For edge applications/devices, you can also use MQTT over websockets: https://www.rabbitmq.com/docs/web-mqtt

Edit: FWIW, NATS also supports MQTT: https://docs.nats.io/running-a-nats-service/configuration/mq...

mrbluecoat• 2 years ago

Any pros/cons compared to EMQX, NanoMQ, or FlashMQ?

phishhook2• 2 years ago

Would be keen to know if this passes Jepsen style tests now

depr• 2 years ago

RabbitMQ is developed by VMware which was acquired by Broadcom. I hope they will remain unaffected.

PHGamer• 2 years ago

interesting so there is no more free support for rabbitmq is what im seeing here for the most part.

okr• 2 years ago

The linked github page gives me 404. But here are the release notes:

https://github.com/rabbitmq/rabbitmq-server/blob/main/releas...

justinclift• 2 years ago

Looks like 4.0.0 was pulled and replaced with a 4.0.1 release.

tony-allan• 2 years ago

https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4....

BurnGpuBurn• 2 years ago

Link without description or anything, nice. After reading the website for 30 seconds, my question would be: What is RabbitMQ?

bdcravens2• 2 years ago

You could say the same thing of any product update link, like the latest versions of Rails, Postgresql, Mongodb, etc. A general assumption is that there are many products that the community is already familiar with, so that an introduction is unnecessary.

RabbitMQ 4.0

Related Stories