Ask HN: What do you use Celery / RabbitMQ for?

https://news.ycombinator.com/item?id=14400073

by zabana • 9 years ago

5 2 9 years ago

news.ycombinator.com

malux85• 9 years ago

To easily distribute jobs over a cluster.

Running right now:

I have a database of 100,000,000 molecules, and I need to compute the force field on all of them. So I put all of the IDs in the celery queue, and then start a worker across 6 nodes,

The job pulls the row from the database, computes the force field, and then stores the result.

Lots of jobs like this: - I used to use it to tune hyperparameters of shallow models - Used it to batch convert a bunch of compressed files from one format to another - Use it with celery beat to schedule scraping of URLs

gerenuk• 9 years ago

We have written distributed crawler using celery/Rabbitmq.

Analyzing social signals real time and it is processing more than 10 million requests everyday. On the way to scale it to process more than 50-100 million requests per day.

Other than that doing nlp tasks along with machine learning using topic modeling, ner etc.

Ask HN: What do you use Celery / RabbitMQ for?

Related Stories