Wield Python's super() like a Jedi

http://rhettinger.wordpress.com/2011/05/26/super-considered-super/

by raymondh • 15 years ago

141 63 15 years ago

rhettinger.wordpress.com

lza• 15 years ago

It's a super() good overview of super(). Nice to see a post that uses a real example.

mclin3• 15 years ago

Have to deal with enough languages as it is. Don't want to further add to my brain mess by learning Python 3 syntax just yet.

raymondh4• 15 years ago

super() was introduced in Python2.2 many years ago. Everything in the article works with very old versions of Python.

Here's a link to the Python 2 version of the examples: http://code.activestate.com/recipes/577721-how-to-use-super-...

kunley• 15 years ago

Python 2.x doesn't have parameterless super() so apparently most of the article's code doesn't work with old versions.

bendotc• 15 years ago

super() was introduced in CPython 2.2, but it changed in Python3: http://www.python.org/dev/peps/pep-3135/. The link you posted uses super as "super(cls, instance)", but the main article uses Python 3's super, which can just be called as "super()", and it figures out the class and instance.

So no, super really has changed, and the syntax in the article does not work with very old version of Python.

mclin3• 15 years ago

super(myclass, self) was. Like I posted below, sometimes I have real trouble keeping the 5 or so languages I use straight. The other day I actually forgot what 'None' is in python because I hadn't used it in a few days, and None is different than in the others.

steve-howard• 15 years ago

Python 3 was introduced two and a half years ago. Its major syntactic changes, while frustrating if you're set in your ways, are hardly insurmountable.

Wickk• 15 years ago

This. Most of the time I run into an issue with a change I wasn't aware of, a quick search through the 3.X docs just points me in the right direction.

mclin3• 15 years ago

I think it's fine if you're doing python all day long, but I'm using Python, Java, PHP, Obj-C and Javascript each at least once a month on a variety of paid and personal projects. Like I said, I have enough trouble keeping those syntaxes straight.

exDM69• 15 years ago

Ah, the "call super" antipattern revived.

__rkaup__• 15 years ago

How's it an antipattern?

veyron4• 15 years ago

how does the runtime performance of super() compare to the hardcoding method? (from your example, using dict.__setitem__ as opposed to super().__setitem__ )

alextp2• 15 years ago

I don't think it suffers much as the method resolution order (mro) is computed ahead-of-time when creating the class.

veyron4• 15 years ago

so super() is computed ahead of time? That doesn't make sense when you look at his comment:

"The calculation depends on both the class where super is called and on the instance’s tree of ancestors."

I agree that the class's MRO can be computed ahead of time, but theoretically the instance's order could be different (if a class method changed the definition of __setitem__)

raymondh4• 15 years ago

In CPython, calling super() has about the same cost as calling any other builtin like len() or int(). The MRO itself is precomputed and stored in inst.__class__.__mro__.

Contrasting dict.__setitem__ with super().__setitem__, the latter adds one function call. In addition, both forms require a builtin lookup and an attribute lookup.

In PyPy, much of the overhead of builtin lookups, function calls, and attribute lookups is automatically optimized away.

kingkilr2• 15 years ago

Unfortunately on PyPy super is less efficient than refering directly to the parent classes method, however it's fixable (and will be fixed).

kalelias3• 15 years ago

tl;dr

Just looks like super/parent like in other languages. Did i miss something from not reading this?

masklinn• 15 years ago

> Did i miss something from not reading this?

Everything? Mostly the part where, Python being an MI language, a given class A has multiple parents. And these parents are a sequence, not a set, which define a Method Resolution Order, and that method resolution order usually isn't breadth-first either, so Python's super is a graph walker. And because it's explicit, any node in the graph can stop the walking.

This means permuting two parent classes can have significant impact on the semantics of the graph walking, therefore the results of calling super.

kalelias3• 15 years ago

Thanks for the short explanation. I totally forget about python being MI.

pdubroy• 15 years ago

Yes.

Peristarkawan12• 15 years ago

Yes, Python's super is more like Dylan's next-method than, say, C#'s super.

silentbicycle2• 15 years ago

Some small steps in the direction of a meta-object protocol: more fine-grained control over call dispatch in a multiply-inheriting class hierarchy.

smosher• 15 years ago

You didn't miss as much as others seem to think. Nothing here changes the fact that it subverts the object model and overuse causes kudzu.

kalelias3• 15 years ago

Yeah, OO is too hard.

guygurari• 15 years ago

It seems to me that code that uses super() in these creative ways will be very difficult to maintain. You need to understand the subtleties of the MRO just to find out which method is being called. Pitty the poor Python programmer who stumbles into such code without being aware that super() may not call the class's direct base.

I suppose similar criticism can be leveled against many other dynamic techniques, except that this one sacrifices more readability than I'm comfortable with.

jessedhillon3• 15 years ago

super() supports arguments which allow you to specify which class's implementation to lookup, which is an explicit way to resolve ambiguities. Although well-designed multiple-inheritance should try to avoid those ambiguities in the first place -- a class shouldn't inherit from multiple bases with conflicting implementations of a method.

I think in a case like the one you describe, where the direct ancestor's implementation is circumvented to access a different ancestor's implementation, it should be done by calling

  super(Ancestor).someMethod()

That should improve understanding.

Peristarkawan12• 15 years ago

I disagree, that's extremely fragile. It might work for the particular class you are implementing. However, it might cause subclasses of that class to break, since those subclasses will have a different MRO than the current class, and you could end up skipping more of the MRO than you intended.

jamesli• 15 years ago

Am I missing some points here?

It seems that the examples are tightly coupled and favors more on inheritance than on composition.

mrshoe• 15 years ago

Personally I'm not in favor of enforcing such strict conventions, especially in a dynamic language like Python. Also isn't this pattern he repeats throughout:

    class Shape:
        def __init__(self, **kwds):
            self.shapename = kwds.pop('shapename')
            super().__init__(**kwds)

Better written like this?

    class Shape:
        def __init__(self, shapename=None, **kwds):
            self.shapename = shapename
            super().__init__(**kwds)

It seems like there are some things about Python's which the author doesn't like, and he solves them by adding a lot of boiler plate to each of his classes. It's probably a better idea to just embrace the Python way or use a different language.

teh• 15 years ago

The issue with your second version is

  In [1]: def init(shapename=None, **kwargs):
     ...:     pass
     ...: 

  In [2]: init(shapename='circle', **{'shapename': 'circle'})
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  
  /home/teh/Desktop/<ipython console> in <module>()

  TypeError: init() got multiple values for keyword argument 'shapename'

Peristarkawan12• 15 years ago

That's entirely the caller's fault, not the function's. Compare:

  >>> def init(**kwargs):
  ...   pass
  ...
  >>> init(shapename='circle', **{'shapename': 'circle'})
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: init() got multiple values for keyword argument 'shapename'

narm60• 15 years ago

These two aren't equivalent, the first example requires the shapename keyword arg, raising a KeyError when it wasn't specified

Peristarkawan12• 15 years ago

That's true. The second one might be better written as:

  def __init__(self, shapename, **kwds):

Or, in Python 3:

  def __init__(self, *, shapename, **kwds):

(making shapename a required keyword-only argument)

mixmastamyk• 15 years ago

Haha, the author is one of the main contributors to python, not some high-schooler.

  kwds.pop('shapename', None)

Will get you what you're looking for.

Peristarkawan12• 15 years ago

It seems to me that the example of combining built-in dictionary classes is naively optimistic. For starters, OrderedDict, as it happens, does not use super! It calls the dict super-class methods directly. Since dict happens to be the next class in the MRO, this doesn't really matter for the purpose of this example, but I can envision a scenario where some plucky programmer inherits from both OrderedCounter and some other dict subclass, and the result doesn't work because OrderedDict does the wrong thing.

And OrderedDict isn't the only one. Maybe for some reason I would like to have an OrderedCounter where all the counts default to 42. So I do this:

  class DefaultOrderedCounter(defaultdict, OrderedCounter):
      pass
  doc = DefaultOrderedCounter(lambda: 42)
  doc.update('abracadabra')

Which results in:

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "c:\python32\lib\collections.py", line 507, in update
      _count_elements(self, iterable)
    File "c:\python32\lib\collections.py", line 63, in __setitem__
      self.__map[key] = link = Link()
  AttributeError: 'DefaultOrderedCounter' object has no attribute '_OrderedDict__map'

Whoops! Apparently defaultdict doesn't use super either. Of course a better way to do this would be to subclass DefaultOrderedCounter and just override the __missing__ method by hand, but that's not the point.

The article goes into "How to Incorporate a Non-cooperative Class", which basically says "wrap it up in a proxy class". But that's not really going to work here, since the result would be two separate dicts, with the defaultdictwrapper methods operating on one dict, and the other methods operating on the other.

Peristarkawan12• 15 years ago

Okay, I give. What's the markup for posting code snippets on this site?

lmkg• 15 years ago

Put two spaces at the beginning of a line for that line to be displayed monospaced.

  Example

simcop2387• 15 years ago

why would you have to subclass DefaultOrderedCounter again in order to add in the missing method?

Though I agree there's something annoying going on there with the multiple inheritence.

Peristarkawan12• 15 years ago

I mistyped. I meant "subclass OrderedCounter to add in the __missing__ method".

lachenmayer• 15 years ago

Nice article, but one thing struck me when reading the code examples posted in the article: What's the use of using kwds in __init__?

    class Shape(Root):
        def __init__(self, **kwds):
            self.shapename = kwds.pop('shapename')
            super().__init__(**kwds)

    class ColoredShape(Shape):
        def __init__(self, **kwds):
            self.color = kwds.pop('color')
            super().__init__(**kwds)

I would personally just have written this somewhat similar to this:

    class Shape(Root):
        def __init__(self, shapename):
            self.shapename = shapename
            super().__init__()

    class ColoredShape(Shape):
        def __init__(self, shapename, color):
            self.shapename = shapename
            self.color = color
            super().__init__(shapename)

But then again I'm by no means what anyone could call a Python 'Jedi'. Can anyone explain to me why using kwds in this way would be better? Would it not lead to having to look up the declaration of the class for the exact names of arguments to pop() every time?

Peristarkawan12• 15 years ago

The reason for using keywords is that there is no guarantee that the next class in the MRO after ColoredShape will be Shape. When the instance is a ColoredShape, it will be, but if ColoredShape is subclassed with a more complex inheritance structure, then something else could end up in between them in the MRO.

Thus, ColoredShape.__init__ can't simply pass on shapename to the super method, because that might not be the argument expected by the next __init__ method in the MRO. Instead, each method needs to gracefully accept arbitrary arguments and then pass on whatever it received (minus what it consumed).

silentbicycle2• 15 years ago

This is a particularly unnecessary title change.

oinksoft• 15 years ago

A true Jedi favors composition.

uriel• 15 years ago

One of the reasons I'm starting to love Go more than Python.

In Go something like super() is superfluous.

jessedhillon3• 15 years ago

The only reason why you don't need super in Go is because you cannot embed multiple interfaces except if their sets of methods are disjoint. This eliminates super's job of allowing programmers to differentiate between base classes with conflicting method implementations.

Without this, Go requires you to design your class hierarchy in a way that is cleaner, but the super concept is still not superfluous -- because you do sometimes need to reference an ancestor's implementation in Go.

You embed an interface and then explicitly reference the embedded interface by name. The Go interface

  type ReaderWriter interface {
    Reader
    Writer
  }

is exactly like this Python

  class ReaderWriter(Reader, Writer):
    pass

To get at a particular superclass implementation, you pass the class to super, like so:

  def read_and_write(self):
    super(Reader).read()
    super(Writer).write()

which in Go is:

  func (rw *ReaderWriter) ReadAndWrite() {
    rw.Reader.read()
    rw.Writer.write()
  }

So it's not superfluous -- there's an equivalent syntax to super which involves referencing embedded interfaces.

raymondh4• 15 years ago

Good advice and well spoken :-) That being said, composing a number of object classes also involves some measure of cooperation, caller/callee ordination, and some plan for calling the methods in the correct order.

IOW, if the decomposition of the classes is the same, then the solutions using composition or using multiple inheritance will have much in common.

Things should be as simple as possible, but no simpler :-)

draebek• 15 years ago

I didn't see it linked in the article nor mentioned here already but several of the problems with `super` have been discussed in "Python's Super is nifty, but you can't use it"[1] for quite some time.

[1] http://fuhm.net/super-harmful/

mikeklaas• 15 years ago

That article is about Python 2's super, which is a different beast. Lessons were learnt in the design of Python 3's version.

Peristarkawan12• 15 years ago

Not really. The only thing that changed about super in Python 3 is that you can call it with no arguments, which is just a short-hand for the way it was already used in Python 2. Everything in that article applies equally to both Python versions.

fferen• 15 years ago

The "kwds" technique is actually something I came up with independently to solve a different problem when I was using PyGame: ridiculously long argument lists when constructing sprites. In this case it significantly increased the flexibility and brevity of the code, although perhaps with a more complicated inheritance tree it wouldn't be worth it.

Nice to see this pattern being acknowledged somewhere else; I thought I was the only one that did that.

raymondh4• 15 years ago

It's a great technique. I'm sure your PyGame code benefitted in a big way :-)

Wield Python's super() like a Jedi

Related Stories