Gearman, an open source framework for distributed processing

by fosk • 15 years ago

40 13 15 years ago

zrail2• 15 years ago

My problem with Gearman is that the queues are completely opaque. There's no way to inspect them, except in so far as if you use a database backend you can see the opaque jobs and get a count.

jbert• 15 years ago

I don't think you can see individual job data, but you (and/or your monitoring daemon) can telnet to the port and issue admin commands. e.g. 'status' will list all job types, with #queued, #running and #workers.

http://gearman.org/?id=protocol (search for 'Administrative Protocol' - 2nd hit)

strags• 15 years ago

How does this compare to resque? (Ah... answered my own question - resque seems to be ruby only).

zrail2• 15 years ago

In short, Gearman is multi-language, multiple different backends, possible to do RPC-style synchronous processing. Resque and friends[1] are single-language, exclusively async, but also extremely transparent because they're just JSON blobs in redis. Resque has a huge following as well.

[1] https://github.com/defunkt/resque/wiki/alternate-implementat...

gizzlon• 15 years ago

Doesn't Gearman do async as well?

evmar• 15 years ago

It's funny to see how bradfitz's projects other than memcached have fared over time. Didn't realize this one was still used by anyone.

jemeshsu• 15 years ago

Can never count Gearman out, and it is winning new user such as Instagram. Great for simple work queue. I recently switch from Celery to Gearman to reduce headache.

dpres• 15 years ago

I was just using Gearman::Client and Gearman::Worker today... it's still used by at least me.

enigmabomb• 15 years ago

We tried to use this in our last project and ended up writing the mother ship and workers from scratch because there didn't seem to be a community behind gearman. As soon as the nda runs out, I'm going to open source it as a competing framework.

rubashov• 15 years ago

What was your problem with it? I have tried to use Condor for a couple things recently and was annoyed enough by it I concluded Gearman was probably the better approach, though I haven't used Gearman.

For the record, Condor probably works fine for its primary use case, which is long running single threaded processes on Unix that eat an input file and spit out an output file before terminating. I was trying to use it to have a bunch of processes on Windows that also occasionally write to a database. Was also trying to tag "slots" on specific machines to use certain installed dependencies. I advise against trying to use Condor and Windows, or on trying to use any of its advertised advanced features beyond the basic "standard" or "universe" model. Have had no end of trouble with the "stored credentials" feature that lets you run a process under a specific account.

mst• 15 years ago

I think a part of the problem there may be that a substantial percentage of the original user community are perl hackers, and tend to discuss their Gearman usage in the context of whatever else they're using - I see Gearman discussions reasonably often on the Catalyst IRC channel, for example.

Another reason is quite possibly that for the vast majority of tasks, Gearman "just works" - if the obvious way to use a piece of software works first time, talking to other users isn't particularly necessary (there's not really a GNU cat community either, for example :)

I'm not entirely sure what you'd do to fix that though (or even if there's anything except perception to be fixed)

pan69• 15 years ago

Tried it about a year ago. Couldn't get it installed on Ubuntu.

maxklein• 15 years ago

I used gearman to replace cron. It's much more efficient, particular for tasks like queueing email to be sent out.

Gearman, an open source framework for distributed processing

Related Stories