Message Queue Evaluation Notes

http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

by wave • 15 years ago

27 11 15 years ago

wiki.secondlife.com

pavlov2• 15 years ago

Can someone more familiar with these products explain why they decided against ZeroMQ even though it apparently offers by far the best performance?

The ZeroMQ "note" part sounds like there's a bug in the Python library, but that hardly seems to be grounds for eliminating it completely from the "race".

rdw3• 15 years ago

I'm the primary author (formerly Which Linden). Surprised to see this pop up here after all this time!

We wrote this almost three years ago, and at the time ZeroMQ was substantially less mature than it is now. In particular, the Python bindings were broken, weren't maintained by the ZeroMQ organization, and there was no indication they'd be production-ready by the time we needed it (which would have been within the next month or two).

More importantly, we were looking for something that handled our use case a little bit more fully. Our message-rate requirements were laughably small, so what we really wanted was software that gave us an "inbox" for each of millions of users. We also weren't looking forward to implementing failover, consistency, and sharding from scratch.

Edit: corrected timeline.

pavlov2• 15 years ago

Thanks for the explanation!

DEinspanjer• 15 years ago

Do you know what they ended up settling on?

fzzzy• 15 years ago

Hi rdw! What are you up to now? I don't have any contact info for you and haven't seen you on irc in forever.

(You might not recognize this nick, this is Donovan)

orenmazor• 15 years ago

there is seriously something to be said for the complete solution that other systems provide.

if only as stepping stones towards eventually using something like zmq.

zacharyvoase• 15 years ago

N.B.: this page wasn't updated heavily since April 2009; a slight revision was made in January 2010.

northisup• 15 years ago

+1 to this, a lot has changed since this was written.

lazy_b• 15 years ago

I'm curious what the motivation for the potential "hundreds of millions" of messaging queues is here. having a queue per-registered user seems, I don't know, maybe overkill? Per group chat makes a lot of sense, but I'm having trouble reconciling the numbers based on what was posted. What am I overlooking?

rdw3• 15 years ago

Second Life involves a lot of per-user message sending. IM, group chats, friend requests, marriage proposals, dialogs with in-game merchant systems, payment notifications, persistent announcements, and a lot that I'm forgetting. They're all roughly the same thing -- messages that need to be delivered to either a specific user or a group of users, with flags for whether they get discarded when the user is offline or whether the user has to explicitly acknowledge them.

It was probably overreaching for us to try to find a system that would handle all this for every user. But even hundreds of thousands of current users, the bare minimum we could expect to be concurrent, were not well supported by the existing solutions.

Message Queue Evaluation Notes

Related Stories