DisRedis: Python Redis client with Automated Sharding and Failover

http://engineering.custommade.com/introducing-disredis-an-open-source-redis-client-to-automate-sharding-and-failover/

by mlader • 13 years ago

14 2 13 years ago

engineering.custommade.com

mrchilds• 13 years ago

I'm one of the developers of DisRedis and the devops engineer running the servers. Let me know if you have any questions.

siliconc0w• 13 years ago

It looks like you're using SHA1 as your hash algorithm which isn't a great choice for sharding. Look into using a consistent hashring algorithm. Basically you want to minimize the number of keys you need to redistribute if you add/remove nodes. https://pypi.python.org/pypi/hash_ring/

What would be cooler than this would be to port the CRUSH algorithm to python. CRUSH is what Ceph (distributed file system) uses to map data to storage servers. This allows you to define a 'map' of how you want data distributed - i.e by rack, row, or datacenter and it can handle replicas, failures, and overloaded systems as well. Whitepaper: http://ceph.com/papers/weil-crush-sc06.pdf

DisRedis: Python Redis client with Automated Sharding and Failover

Related Stories