How to be below API rate limits from multiple servers

https://callhub.io/blog/2014/02/03/distributed-rate-limiting-with-redis-and-celery/

by ad93611 • 12 years ago

28 5 12 years ago

hox• 12 years ago

Interestingly enough, Alan Shreve gave a talk at RedisConf a couple of years back detailing something very similar used on the twilio stack to rate limit draining a queue: https://inconshreveable.com/talks.html

akerl_• 12 years ago

Reading the title, I was expecting to see a somewhat shady article on using multiple servers to dodge API rate limits.

Instead, they've got a pretty snazzy writeup of how to effectively keep your distributed processes working together properly. I'm curious why they didn't pass a token rather than the actual Lua over the wire via Redis, but it certainly seems to work for them.

wheaties• 12 years ago

So they send code to Redis to make sure that they won't increment too high? I know Redis already has a ton of functions prepackaged but this seems like it would be a good one: INCRIF.

rryan• 12 years ago

I may be missing something but I think this could be done more cleanly with an atomic counter or semaphore [1] in Redis.

  1. Have a task that releases 1 resource to the semaphore    
     every (1.0 / cps) seconds.

  2. Workers wait to acquire a resource from the semaphore 
     before making a call.

Due to the rate of release being fixed at 1/cps no worker can ever exceed the calls-per-second limit. Much simpler than sending Lua over the wire.

[1] https://github.com/dv/redis-semaphore

erikcw• 12 years ago

The only downside I see to this approach in the context of this post is that then you'd essentially need to dedicated daemon sending releases to the semaphore for #1.

I suppose you could use a long running celery task for that purpose. You wouldn't want to fire of (1.0 / cps) celery tasks just for releasing semaphores unless the value of cps was very small -- as it would congest your queues with a bunch of small cleanup tasks. Which aren't guaranteed to run precisely that often.

How to be below API rate limits from multiple servers

Related Stories