Introducing Disque [video]

http://www.thedotpost.com/2015/06/salvatore-sanfilippo-introducing-disque

by sylvinus • 11 years ago

99 32 11 years ago

thedotpost.com

antirez9• 11 years ago

$ ./antirez --voice "godfather" --talk

(I hope is understandable at least...)

ie2• 11 years ago

Understood everything no problem. Having recently implemented a job queue system using Redis I am excited to see Disque.

I wonder if it makes sense to use Disque in the same environment as Redis, since we have use cases for both a key-value store as well as a job queue.

antirez9• 11 years ago

Thanks, definitely makes sense to use Disque for messaging and Redis for caching/storage, and for messaging too when you need ephemeral messaging without guarantees, but you need to control very well the semantics, so in Redis you can build your messaging thing using its primitives. But for reliable message exchanges, with guaranteed at least once delivery, Disque is the thing to use (or other good & reliable message queues).

linkmotif• 11 years ago

So nice to see you post this here. While watching I was really enjoying your Italian accent and was thinking about how amazing it is that so many people have to learn English. Reminded me of your post "English has been my pain for 15 years" (http://antirez.com/news/61). Thank you for conquering the language and presenting this to us. Really glad to have another message queue to explore.

antirez9• 11 years ago

Thanks linkmotif, when I finished talking, as usually, there were people literally ROTFLing for the accent, but in a positive way. Finally at least I can kinda talk in a more casual way, it used to be harder than this in the past because of totally lack of practice, usually I just write/read. Thanks for your message :-)

joeskyyy• 11 years ago

I've never had issues understanding you before man! Always amazing to hear your thoughts out loud because it shows how much you truly care about the projects you put so much time in :D

guiomie• 11 years ago

Looks interesting. But why would I use this instead of RabbitMQ ? (I didn't watch the video, but I did check the github repo)

matthewmacleod2• 11 years ago

I reckon it'll be a complexity thing.

Disque uses the same kind of tremendously simple API as Redis does. It appears to have ~15 commands, and is basically just a job broker and nothing else. If it gains traction, it will solve a specific common problem well.

RabbitMQ is a more comprehensive message broker, and more complex. It ships with federation, an HTTP API, SASL, that sort of thing. Useful tool, but likely more complex to setup and administer (to the point that it might be excessive for some use cases).

Obviously this is all subject to personal suitability though!

shockzzz• 11 years ago

I like Salvatore.

sehr• 11 years ago

I'm almost ashamed I didn't know his real name before, but I guess his projects speak for themselves!

alphadevx2• 11 years ago

How does this compare to Resque (I've used that in production and like it a lot)?

antirez9• 11 years ago

Resque is based on Redis, so it is similar but can't offer the same guarantees, scalability and fault tolerance. However for many use cases where ephemeral messaging is enough, Resque is a good pick.

alphadevx2• 11 years ago

I love Redis and I love messaging systems, so will definitely take a hard look at Disque, thanks for all your hard work!

matthewmacleod2• 11 years ago

I'm pretty excited – I've spent a couple of weeks on and off trying to build a useful reliable job queue system on top of Redis, but it's not really working as simply as I'd like.

Disque seems to offer the features I am interested in along with the relatively simple semantics of Redis. Looking forward to the API being settled a bit so I can play with it.

tempVariable• 11 years ago

Thank you for providing yet another project to the open source community Salvatore.

antirez9• 11 years ago

Thanks! My pleasure, and I'm already receiving tons of help from the community.

yazaddaruvala• 11 years ago

antirez as you mentioned both projects (Redis, Disque) are based on the same "framework" (skeleton).

Do you think because of their different guarantees that shared code will diverge?

Do you think that base layer will have to be kept in sync? (bug fixes, improvements)

I guess I'm asking, if at all, at what point would you pull that skeleton out into a separate repo and share it across Redis and Disque?

antirez9• 11 years ago

Initially I had to face this choice, and I decided to go for the independent fork for a couple of reasons:

1) As you said, my guess was that the core part would diverge dramatically, if not for the separated files that can be kept in sync anyway, like event loop, string library, and so forth. And the time is showing this assumption was true. The code is diverging and as it diverges the effort to take it under the same library may become complex.

2) This was even important than "1": the effort to rewrite Redis in terms of a common core was a big one. I was not able to stop the development to undertake this effort, just for a side project which was not clear at the time where was headed. Now I'm more confident that Disque will get my attentions for the next years, but still I don't have the time to attempt such a big refactoring / rewrite.

However I cherry-pick from the common parts, and do some "diff" from time to time to see if there are ideas in one side that can be merged in the other side. Is not optimal, but I had no choices. What can be done in the future is to turn more parts of both systems into common libraries that can live in separated files, and rewrite only those parts, incrementally.

yodon• 11 years ago

For anyone curious, Disque is a synchronously replicated in-memory message broker/job queue that was designed to capture the essence of the "Redis as a jobs queue" use case.

arturhoo• 11 years ago

Hi Antirez, always nice to use and see work from a good professional and outstanding community member. Whenever I saw people using Redis as a queue server, it was mainly for two reasons: 1) they were already using Redis and it had the right primitives, 2) it was dead simple to setup and use (compared to rabbitmq, for example).

One guy that I think is often overlooked is beanstalkd (https://github.com/kr/beanstalkd), especially with regards to 2). Have you ever taken a look at the project and if so, how do you think it compares to your vision for Disque?

antirez9• 11 years ago

As the author of one of the two systems I don't feel super well at comparing Disque and Benstalkd, but I guess, as Disque matures, people informed about both systems will compare them. However an obvious difference is that Disque is "internally" distributed and federated, while Benstalkd requires the client to take care of managing multiple nodes.

nodesocket• 11 years ago

Does Disque support recurring jobs, like queue this task, and run it once every 4 hours. I know Antirez mentioned there is a delay property in the video.

Also will there be support for absolute scheduling like, run once at 4:30:00am today.

waxjar• 11 years ago

A distributed cron job cluster. Interesting!

rakoo• 11 years ago

What more would it have than a simple script in a crontab that pushes the job every X ?

bluecmd2• 11 years ago

I find it interesting that the article claims eventual consistency, but also best effort ordering. The consistency in a queue is surely made of two parts: count and order. To not have duplicates is nice, but if ordering is not guaranteed - is this really a queue? Seems like it's a Something In Something Out, or "a pile" or something. Again, curious about what the evental consistency guarantee is referring to.

Not saying it cannot be useful - for work distribution it will surely work - but it needs to be considered for your usecase that you will need to be able to process messages in an unlimited unordering. Process a delete action before a create for example.

antirez9• 11 years ago

What is eventually consistent is the job state: it means that jobs are removed only when they are successfully delivered at least one time.

Best effort ordering does not mean no ordering btw. Use cases for which strict ordering is needed are rare, and it is possible to build order out of an unordered queue if strictly needed (especially since, when strict ordering is needed, usually you have a source of truth to maintain the state of the application, that happens to be a strictly consistent store).

The cost of ordering is huge in terms of design, and benefits just an handful of use cases. So I think best effort ordering is one of the best Disque assets.

bluecmd2• 11 years ago

Ah OK. This might sound harsh, but so the guarantee you have is that the data will not be lost? That's a pretty weak guarantee no?

Also, best effort implies no ordering in the worst case no? I always try to judge systems when the world isn't peaches.

mattkrea• 11 years ago

As someone who has to maintain a RabbitMQ cluster.. I cannot wait to be able to use this reliabily.

Introducing Disque [video]

Related Stories