Machinery: An asynchronous task/job queue for Golang

https://github.com/RichardKnop/machinery

by richardknop • 11 years ago

26 10 11 years ago

github.com

rw• 11 years ago

Badly titled. This is not an official library.

richardknop5• 11 years ago

I'm sorry. I changed the title.

bsaul• 11 years ago

Great ! I was just starting today to think about how something like that could be done in golang.

I think a full example should involve the client part as well ( in order to remain typed along the chain, and have the json format describing the task an implementation detail).

Edit : i didn't read properly, the readme also describes how a task is created. I didn't expect it to use a low level format though. Why not relying directly on the type of the task ( "AddTask") rather than asking for a "add" string parameter ?

richardknop5• 11 years ago

There is a very basic client implementation of client in examples/send/send.go.

This is a very early stage project though. I agree that some parts of the API might need to be rethought and other approaches considered.

jzelinskie• 11 years ago

Looks like a good start, but from a quick glance it is missing some Go idioms. Some cursory recommendations:

- No Makefile please, this project's build process isn't yet complicated enough to warrant one.

- Rename the App type something more descriptive like Server

- Rename InitApp to NewApp (NewXXX is an idiomatic Go constructor). Likewise with InitWorker

- Make NewWorker a method of an App

- Isolate the AMQPConnection into another library and use the design pattern in database/sql to allow users to create their own implementations of brokers

- Do whatever you can to avoid empty interfaces (aka interface{}). You may have to be clever, but really avoid these at all costs. Using them basically mentions you'll have to sprinkle type assertions everywhere in the code rather than maybe having one place where you unmarshal values.

richardknop5• 11 years ago

Thank you very much for advice. Yes, I agree empty interfaces{} are a bad way of approaching this.

I am thinking of using reflection in order to achieve this. I will need to think of some clever way to describe the types in the message so I can then use reflection to unpack JSON into proper types and pass them to a task (without making the API too overcomplicated). I will try to do this over next couple of weekends. Hopefully I can come up with some clever and clean API.

Isolating AMQPConnection is also a great idea, I am planning to support multiple brokers (Redis, database etc) in the future.

richardknop5• 11 years ago

I did a quick implementation of reflection today so arguments are not empty interfaces now and tasks are just simple functions with properly typed arguments. I will think more about simplifying it further.

armon• 11 years ago

This looks pretty interesting! We use a library we've built called relay (https://github.com/armon/relay) to do something similar also in Go.

I like the higher level task based approach of this library, since this is a pretty common pattern when using task brokers.

richardknop5• 11 years ago

Looks very interesting. I will look over the code in next couple of days :)

jaytaylor• 11 years ago

How is this different from a simple async processing queue with Kafka?

Machinery: An asynchronous task/job queue for Golang

Related Stories