Easy cluster parallelization with ZeroMQ

http://mdup.fr/blog/easy-cluster-parallelization-with-zeromq

by mdup • 11 years ago

64 18 11 years ago

gh02t4• 11 years ago

Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ as the backend as well. If you've got a small to medium sized task (I'd say less than 500 nodes is a good number) to run on your cluster ipcluster is pretty great. Even when I have a proper queuing system like SLURM or PBS in place, I still often use ipcluster on top.

http://ipython.org/ipython-doc/dev/parallel/

nickpsecurity7• 11 years ago

Thanks for the link. That might come in handy if I do any more clustering.

andreabedini• 11 years ago

I'm curious on how you use ipcluster on top of pbs

gh02t4• 11 years ago

If you mean the practicalities of how to use it, they describe it a bit here http://ipython.org/ipython-doc/dev/parallel/parallel_process... .

If you mean how have I used it specifically, I've used it to do a variety of things. Ranging from recreating what was described in the original post to using it as an ad hoc MPI replacement. I find it's handy for managing jobs that have some coupling (and so aren't easily suited to running via the features of the batch system), but aren't tightly coupled enough to warrant the suffering of having to write MPI-based code.

Merkur• 11 years ago

I had fun reading - thx. I found the solution of "requesting workers" interesting. My first reflex to the round robin argument was - a better load balancer but the requesting worker made that totally not necessary! I wounder what negative sites sutch architectures may have?

nickpsecurity7• 11 years ago

Nice article. I hope to see a benchmark of it on a less parallel problem with low-latency network hardware. Maybe a comparison of that against a vanilla MPI or parallel framework. We'll get to see what its real overhead is.

gh02t4• 11 years ago

It's a bit hard to compare to MPI, which is considerably lower level and is oriented towards tightly coupled compute clusters. MPI is more about passing around data structures and not so much about dispatch and execution of independent tasks, which is more of a distributed computing type endeavor. You can build something like this on top of MPI (and I did once as an experiment), but it is tough to compare them directly since they are designed on different assumptions.

nickpsecurity7• 11 years ago

Fair point. I've been out of cluster computing for years so what do you think would be comparable?

avargas• 11 years ago

ZeroMQ is amazing. A few years ago I built a prototype project for a client that basically was fail2ban but scalable. It monitored nginx logs and broadcasted some information, and workers did the rest. Most of the data was in memory and passed around, and communication was done via ZeroMQ. It was done this way so that we could split the heavy-load components off the server and into workers, and allow the server to simply do it's job: tail nginx logs and act upon ban requests from workers. It was amazing, sadly I never completed it and deployed it on a production environment but from initial tests, it outperformed fail2ban by a lot.

FullyFunctional• 11 years ago

Great writeup. Perhaps slightly less clean, you could cut the messages in half by having the available report the result as well (empty in the initial case).

Is the worker idling between replying and receiving new work? In other words, does ZMQ enable overlapping, say always having two outstanding requests per worker?

Easy cluster parallelization with ZeroMQ

Related Stories