A Guide to Python's Magic Methods

http://www.rafekettler.com/magicmethods.html

by cskau • 14 years ago

234 27 14 years ago

MikeOnFire• 14 years ago

For each special method, you should show what syntax magically invokes it. Something like this:

foo.__iter__() is invoked by for x in foo a.__iadd__(b) is invoked by a += b a.__rmult__(b) is invoked by b * a when b * a fails. foo.__len__() is invoked by if foo when foo.__bool__ or foo.__nonzero__ is not defined. foo.__repr__() is invoked by '%r' % foo, also by pprint.pprint(foo) and deprecated `foo`. It's also invoked by __str__ magic when foo.__str__ is not defined. etc. Putting the above info in a table would also be helpful.

rafekett3• 14 years ago

This was my work (not sure why someone randomly posted it here, but such is the internet).

I do exactly what you just described, see http://www.rafekettler.com/magicmethods.html#appendix

cskau• 14 years ago

Such is the Internet indeed.

I stumbled upon this while researching some Python magic. I felt it was a pretty good write-up so I decided to share it with my fellow HN'ers.

Thank you for posting this.

ceol• 14 years ago

I'm quite glad someone posted it here! I'm not sure if I would have found such a useful reference otherwise.

Thanks for writing this up!

masklinn2• 14 years ago

You should not document __coerce__, it shouldn't be used anymore.

Also why put __nonzero__, __str__ and __unicode__ in "Representing your Classes" instead of "Type conversion magic methods"? Why not talk about the collections ABCs for container operators? (using them as mixins greatly implifies implementing containers, and makes it clear what's required and what is not)?

And the __concat__ note is incorrect, that does not exist in the userland datamodel it exists only in the `operator` module and in the C sequence protocol.

jerf• 14 years ago

The word example is dangerous; two different words of the same length are neither less than each other, greater than each other, or equal to each other. I think sorting may not work as well as you'd like, and you should just be passing a custom cmp function to the sort method. The other comparison methods are for optimization; they should never disagree with __cmp__, and if __cmp__ is already optimal you should usually not define them.

masklinn2• 14 years ago

> a.__rmult__(b) is invoked by b * a when b * a fails.

And for boolean comparisons, mirrored is an other operator: if trying a >= b and a.__ge__(b) fails, then the interpreter will fall back on b.__le__(a) (and if that fails as well, then both CPython and Pypy will compare the value's type's name in Python 2 although that's considered an implementation detail — and the interpreters may behave differently as some types have different casings)

Also, if I remember correctly "failure" is either the method not existing or the method returning NotImplemented.

An other interesting fallback is `in` (`__contains__`) which falls back on iteration when there is no `__contains__` and iteration itself (`__iter__`) falls back on `__getitem__(int)` when `__iter__` is not defined.

So you can have `a in b` ending up calling `a.__getitem__(0)`, which the container does not always expect (when it's not been coded correctly)

It's rather sad that TFA does not explain these, or in which case reflected arithmetic operators get called (his example makes very little sense, especially for non-commutative operators).

> foo.__bool__ or foo.__nonzero__

Important note: the former is the Python 3 version of the latter, defining __bool__ in Python 2 does not work.

mitchellh• 14 years ago

This is a great post, but I also want to point out that this page is invaluable to any Python developer wanting to know the details about the magic methods: http://docs.python.org/reference/datamodel.html#special-meth...

Whenever working on any reasonably complex class, I find that open in a tab to verify that I'm using the magic methods properly, and that they'll be called as I expect.

rafekett3• 14 years ago

I agree. What I wrote here was based on that.

I wanted my post to be more akin to a tutorial than a reference. I don't think the data model reference is useful to people who don't know the basics -- the idea was to bridge that gap so that it could be useful.

steve-howard3• 14 years ago

>__getattr__(self, name)

> ... This can be useful for catching and redirecting common misspellings ...

I can think of only one good reason to redirect spelling errors, and that's if you are a widely-used library and you want to support both "color" and "colour" (and similar Britishisms). Other than that, you're just asking for a headache.

chernevik• 14 years ago

I am but an egg . . . and while I agree this stuff is great, it is magic and I think it helps to write code so your usage is explicit. Sometimes all that's needed is a hint that something is Not Normal, so the code reader is alerted to look carefully. A test showing Foo(4) * Bar('grandma') == 'nice' can at least flag and often explain whatever mysteries you've created. If you don't put the tests in doc strings with doc tests, at least put a reference to them in the module so the reviewer knows to look for them.

Stuff like __getattr__ is particularly tricky. I also try to end __getattr__ code with an AttributeException, which I find really helps debugging. I like using this to dispatch calls to objects better built to fill them, but I try to do so explicitly within the __getattr__ code to give reviewers a clue as to what's going on. For example, I'll check the attribute name against a list before passing it on to the target object. Even if I intent to pass _everything_ to a target, I'll first check the target for the passed attribute with getattr(target, attribute name, False), so that failed passthroughs stay with the object calling them. (You can generate messages indicating the problem -- "Can't find 'RIAA' in object 'Nice'" and pass those to an AttributeException.) Ideally any string validly usable as an instance attribute should be findable somewhere in the code by grep.

jules• 14 years ago

It would be perhaps be useful to throw an error saying "X object has no attribute Y, did you mean Z instead?".

steve-howard3• 14 years ago

That's probably a better idea. In either case being "liberal in what you accept" is a discredited software practice.

zanny• 14 years ago

This might be tangental, but the syntax of having magic methods be surrounded by double underscores always drives me insane. For a language designed around readability, it seems like the python dev team just kicked the can on things like __init_ and said "Sure, we have a few dozen english words already restricted usage in the language, but let's kick the whole "pretty" thing out and start wrapping every special statement in ugly underscores like its 1995 again!".

They probably started this trend around 1995, now that I think about it, ya know. since the language is that old and all.

It is still really annoying, I just cringe when I use __init__ constructors because they look so ugly in otherwise very English-document-like source.

</rant>

njharman3• 14 years ago

Special methods are special. It is a feature that they look different and stand out.

batista• 14 years ago

Well, they could stand out with some better say.

How about a "special" keyboard?

def special init()

How about @?

def @init()

or ::

def special::init()

or, all caps, like Go does initial-cap for exported members:

def INIT()

nyellin• 14 years ago

Personally, I love the __names__, as they instantly identify magic methods and there are never naming conflicts. Contrast this with Java where a newcomer might name their own method toString() or readObject() by accident.

Ingaz• 14 years ago

I like underscores for all magic methods.

And I think you're right about __init__ - constructors are not magic.

But in fact, I like "word" __init__ even more outside of python.

I use it to name directories for all new, incoming, uncategorized stuff.

For example: I have "music" with subdirectories named by artist, and I have __init__ where I put all artists I found recently. Same for "books", "docs", "work", etc.

I like it because "__init__" is always on top. Compare it with "new", "incoming", "now", "unclassified".

njharman3• 14 years ago

Controlling Attribute Access section is really poor. Giving poor explanation of getattr vs getattribute, wrongly recommending to not use these methods, claiming there is little use, and providing poor examples of common use cases. The biggest of which is object composition.

jeffjose• 14 years ago

I'm not sure who downvoted you, but I felt the same.

Its been a while since I've been through getattr vs getattribute pages, but I distinctly remember there's a big big difference in how you use them. I was kinda hoping the page had it, but unfortunately it didnt.

Original Author, if you're reading this - kindly do that.

SoftwareMaven• 14 years ago

__getattribute__ (new classes only!) hijacks the entire lookup machinery; every time you write instance., you will call it.

__getattr__ allows "normal" lookup (is it a method, does it exist in __dict__, etc) to be done by the VM and isn't called until you would otherwise get an AttributeError.

sophacles• 14 years ago

One of the better treatments of __get__ and __set__ I've seen. Descriptors are deep and powerful magic, and a bit complex, but when they are appropriate, they are less complex than the alternatives, and allow for cleaner interfaces.

A Guide to Python's Magic Methods

Related Stories