Mixpanel went down, so we're open sourcing a better pure Python memcache client

http://code.mixpanel.com/2012/07/16/we-went-down-so-we-wrote-a-better-pure-python-memcache-client/

97 17 14 years ago

code.mixpanel.com

lamby• 14 years ago

Perhaps it could be named better; all the python memcache clients have almost identical names.

> Because we use eventlet, we need a pure python memcache client so that eventlet can patch the socket operations to be non-blocking.

Not true. ultramemcache (https://github.com/esnme/ultramemcache) is an example of a c++ memcached client that is friendly to gevent's monkeypatching. It does this by importing the socket library via the python runtime (https://github.com/esnme/ultramemcache/blob/master/python/um...), rather than using a native construct.

My experience is with gevent, but I suspect that eventlet's monkeypatching is similar to gevent.

gtaylor• 14 years ago

pylibmc may be fine, as well.

latchkey• 14 years ago

This is the type of response that makes me a happy Mixpanel user. Hit a snag; fix it, talk about it, open source it.

tebeka• 14 years ago

Yet again, instead of trying to fix the existing, someone writes yet another library.

rhizome• 14 years ago

This was my thought, too: there seems to be a limit beyond which memcached simply breaks. I learned this 5 years ago, and it had nothing to do with Python.

catch23• 14 years ago

Well, if there are too many users of an existing library, it makes it hard to do large changes. So in their case, it seems like the original library would silently fail on exceptions, they want to make things bubble up to the top which could be bad for a large number of users who have been depending on the silent-fail operation.

evan2m• 14 years ago

I would argue that python-memcached has been replaced as "the de-facto standard" by pylibmc. I'm not sure if pylibmc works with eventlet or not.

Companies open sourcing code is always good. But, I would be surprised if there wasn't already a python memcached client that would suit your needs.

seunosewa• 14 years ago

I've been unable to compile pylibmc Centos 5.

kapso• 14 years ago

"It turns out that socket operations that fail in python-memcached don’t return an error or raise any exceptions."

...Whats surprising is that this went untested for so long.

suhail• 14 years ago

We've been very well aware of it and safely worked around it at the time. It was also something we just disliked and so we changed it across our entire codebase when we decided to write our own library.

When we said "it turns out", we didn't mean that we had suddenly learned that after failure. It's just something we meant to note that we decided to fix.

jwegan• 14 years ago

You might want to set TCP_NODELAY on the socket to disable Nagle's algorithm (http://en.wikipedia.org/wiki/Nagles_algorithm). We ran into some issues with memcached where if Nagle's algorithm was not turned of, GETs would often take several milliseconds longer than was necessary.

keypusher• 14 years ago

> We enforce the calling convention with code in our library that looks like this:

if not isinstance(val, str): raise ValidationException('value must be str', val)

I love Python, but the more I used it the more I come to see dynamic typing as a mistake. On large projects with shared library code and deeply nested calls, one of the most pervasive problems seems to be wrongly typed data. And building validation exceptions into all of your code seems like a poor solution.

__alexs• 14 years ago

They could have implemented it using a simple decorator for non var-args methods which personally I think is more attractive.

Obviously you still pay the cost at run-time but at least the actual source looks more sensible that way.

bifrost• 14 years ago

I'm confused, why would a cache being down cause data loss?

Two months is also not battle tested, perhaps the terminology used here needs to be adjusted...

mryan• 14 years ago

> Calling .get() on a dead server returns None on failure, the same thing that is returned when there’s no value.

I suppose they were doing a .get() to check if the server was still up, and assuming None meant "there are no values to get, but the server is up". If .set() works in the same way (silently failing) they were essentially using /dev/null as their task queue. :)

illumen• 14 years ago

Yeah. It's a cache, it says so in the name. Why would you make not-a-cache on a cache protocol and library? In-sane.

Mixpanel went down, so we're open sourcing a better pure Python memcache client

Related Stories