Show HN: Martinet – Distributed Task Queue for Node.js

by mpconlen • 12 years ago

28 9 12 years ago

meritt3• 12 years ago

Generally speaking, using a database as a queue is a particularly bad anti-pattern.

mpconlen2• 12 years ago

I guess this should be clarified more in the README - this uses zmq (and potentially other brokers) as the queue, and uses the database for tracking results, saving recurring tasks, etc

meritt3• 12 years ago

Good to know, so more like a crontab entry for the database.

Where's the actual queue though? zmq isn't a message queue, it's more akin to a protocol. e.g. if I have N workers and M jobs, and M>N, where are all the >N indexed jobs kept while the others are being processed?

Like it appears you're using PUSH/PULL pattern, which means the jobs queue up in the publisher's memory (on a per PULL-peer basis..). So if the publisher crashes, those jobs are lost. If the number of jobs queued exceeds the high-water mark, those additional pushes are either going to discard messages or block. If it blocks, it looks like your code is going to have lots of other issues... If the worker crashes, I believe that queue is also going to be lost.

fourstar• 12 years ago

Why?

Redis is a "database" and is used as a queue for many different things. kue.js uses redis for its job queue. Agenda.js uses mongo for its job queue (which has a very similar API to this one listed).

meritt3• 12 years ago

The context was a RDBMS, not Redis. Redis makes a perfectly suitable queue, I agree.

The problem with using a RDBMS as a queue is it results in a continuous polling loop, it's incredibly slow, guaranteeing only a single subscribers gets a message is incredibly complex (you need to employ locking mechanisms yourself), etc.

Using Redis you can just BRPOP and it's perfect.

bhouston2• 12 years ago

Using a database as the backend of the queue is a really good idea. Of course if you have a lot of load you can add a redis cache in front, but persistence and reliability of the queue data is highly important.

If not a DB, how else do you get consistency, reliability and ease of development? Basically you will end up re-implementing a database anyway.

PS. I wrote a highly popular render manager / queuer and I first ended up writing a custom database, but eventually the core was replaced with MongoDB and it is now even more scalable and robust.

bhouston2• 12 years ago

Just use the nodejs api for beanstalks.

fasteo• 12 years ago

How is this different from beanstalkd ? From the README file persistence seems like its main feature, but beanstalkd already has this (via a write-ahead log), along with priority, delayed jobs, burying/kicking jobs, TTR, explicit ACK (for distributed workers), etc.

Show HN: Martinet – Distributed Task Queue for Node.js

Related Stories