A primer on Python decorators

http://www.thumbtack.com/engineering/a-primer-on-python-decorators/

by midas • 14 years ago

187 41 14 years ago

jdwhit2• 14 years ago

Great primer, reading it gives a good sense for why the decorator syntax was made and the potential uses. Is this part of a wider series you are running?

enaeseth3• 14 years ago

Thanks! It is indeed the first in a series of Python posts that will be on the Thumbtack engineering blog.

zackattack2• 14 years ago

Can you please email all future posts to me? ;p zackster@gmail.com -- I don't use RSS any more, and I'd hate to miss them.

DasIch• 14 years ago

There is a small mistake in the post: decorators cannot be arbitrary expressions. Something like @Foo(spam).bar fails with an unhelpful SyntaxError, something that everyone who designs complex APIs will probably encounter at some point. This is a restriction which is in place because Guido didn't like arbitrary expressions as decorators.

enaeseth3• 14 years ago

Oh, Guido. Thanks, I've updated the post.

kelvin0• 14 years ago

Love using Python, but still haven't got around to using decorators yet. I probably didn't understand the concept properly until now ...

gbog5• 14 years ago

You could say decorators are for the code that you want to put inside a function but that don't really belong to its logic. The memoization example shows that: if you have memoization logic inside the function it works but you should feel that there are two disjoint logics at works.

lhnz• 14 years ago

What I really want to know is how a framework such as Flask uses a decorator for the route. How is the correct function picked for a particular route that is defined against the decorator? (Maybe I'm completely misunderstanding this...)

axiak2• 14 years ago

The decorator can do anything. In flask the decorator syntax is just shorthand for adding the route to a registry which is looked up every request.

The source code is pitifully small: https://bitbucket.org/mitsuhiko/flask/src/4d82231621fc/flask...

wahnfrieden2• 14 years ago

Calling a decorator can mutate some shared state.

enaeseth3• 14 years ago

Here's the source of Flask's `route` decorator:

https://github.com/mitsuhiko/flask/blob/master/flask/app.py#...

When you call `route` with a URL pattern, it returns an inner function which is used as the decorator. That decorator just records your route and function in the Flask URL map, and returns your function unchanged.

So, Flask is arguably perpetuating a slight abuse of decorators, since it doesn't decorate or wrap your function at all, but merely saves a reference to it somewhere. But it's a fairly clean way to make up for the lack of code blocks or multi-statement anonymous functions in Python.

gostevehoward2• 14 years ago

Some might say it's a dangerous abuse of decorators.

Since decorators (generally) run at module load time, any stateful decorator (usually) implies the use of global mutable state, which is (considered by many to be) the root cause of much bad design, convoluted flow, limited reusability and untestability. This is perhaps why most decorators in the standard library are pure (off the top of my head).

This is well beyond the scope of the post but an important and often overlooked point in my opinion.

axiak2• 14 years ago

In flask, the app.route decorator mutates the app object. There are no globals necessary. Use of globals to maintain state is an orthogonal issue to use of decorators to update registries.

cdavid• 14 years ago

While you're right it is an easy to make mistake with decorators, I don't think the issue is global mutable state as much as uncontrolled mutable state, especially if used against singleton.

Typically, doing this:

    @register
    def foo():
    ….

is bad, but this is much better:

    @registry.register
    def foo():
    …

if registry is a global object that is not a singleton. In that case, you can easily use distinct registries (for testing, etc…) and this is not much of an issue in practice. Another way of doing this kind of things is to post_pone the registration, with the decorator just labelling things:

    @register
    def foo():
    …

    def register(f):
        return WrappedFunc(f)

and then build an explicit list of modules being imported, and look for every instance of WrappedFunc in that module locals(). I use this in my own packaging project where people can define multiple python scripts that hook into different stages of the packaging.

emboss• 14 years ago

Minor nitpicking:

> unlike in Java, you can also call a class method on an instance

It's possible in Java, too. It's just considered bad practice.

Still, a very nice read!

gbog5• 14 years ago

I'd say if you have to do that, it is most likely because you have a design problem somewhere. It is a code smell in Python as well.

jmurley• 14 years ago

Thanks, but my favorite explanation of decorators is still this stackoverflow answer (see the second answer)

http://stackoverflow.com/questions/739654/understanding-pyth...

ming-hack• 14 years ago

Yes, I think this post is more informative.

Aissen• 14 years ago

Direct link: http://stackoverflow.com/a/1594484/378825

oberon• 14 years ago

For an interesting use of Python decorators - event handling - have a look at Decovent on pypi http://pypi.python.org/pypi/Decovent.

timClicks• 14 years ago

This is similar to how Pyglet does things: http://pyglet.org/doc/programming_guide/hello_world.html

gbog5• 14 years ago

To authors: I would avoid try except in this code snippet, a simple if else is more explicit. I would also avoid a = b = c statement. One line per statement is better most of the time.

samdk• 14 years ago

The Python community generally advocates an "it's easier to ask for forgiveness than permission" coding style. When faced with a condition of the form "if condition a holds, do b, else c", it's very often a better idea to do "let's try b, and do c in case b fails because condition a didn't hold".

In this case it's better because you can avoid computing an extra hash of the object in cases where it's already a key of the dictionary. This may seem like a silly optimization, but it can very easily add up if you're accessing existing elements most of the time--I once had a bit of code that went 60x faster when I replaced an if-else with a try-except.

In other cases it can be even more beneficial. Say you're opening a file. One approach to avoid errors would be to check if a file exists first. This is error-prone because the file might cease to exist in between the 'if' and the 'open' statements, and now you have no code written to handle the error. Using try-except will ensure that you actually handle the error intelligently.

This isn't to say there's never a good reason to use an 'if' to check things, just that if you can do it in one step instead of two, one is usually better.

gbog5• 14 years ago

Agree with that "try except" is better than "if else" for file handling ("with" is even better).

I am not sure if "try except" is really faster than "if else" in some edge cases in a memoization context, as you claim.

What I am sure is that in a didactic context, where you want people to understand code, something like:

    if args not in stored_results:
        stored_results[args] = fn(*args)

    return stored_results[args]

is much better than that (from the OP):

    try:
        # try to get the cached result
        return stored_results[args]
    except KeyError:
        # nothing was cached for those args. let's fix that.
        result = stored_results[args] = fn(*args)
        return result

jmlane• 14 years ago

Out of curiosity, in a situation where you are doing negligible condition testing, is try–except still considered "more performant" than if–else flow control? For some reason (and I'm going to do so reading to get clarity on this point), I have had the silly notion that throwing and handling exceptions can be costly.

locci• 14 years ago

I'm no pythonist, but I think setdefault would be perfect for the case

http://docs.python.org/library/stdtypes.html#dict.setdefault

b0rsuk• 14 years ago

I agree. The author uses exception for something that is not at all exceptional. In fact, it will occur at least once. This is misuse of an exception. No one expects Spanish Exception!

Earlier, he uses the map function for no apparent reason. I mean - list comprehension would perfectly fit here.

I still think I have something to learn from the article, though.

SpiderX3• 14 years ago

The try/except is the preferred python way of doing things. In Python, try/except is cheap, and in the case where the try actually succeeds, you can gain a slight performance benefit. In this case, since performance is the goal, it is perfect for memoize.

gbog5• 14 years ago

I think subtle optimization should be left aside in a tutorial. Try except is a good way to handle many cases that are not exceptions, like when reaching the end of an iteration, but I never heard that it was "the preferred way" for checking the presence of a key in a dict. I believe the "if in" construct is cleaner, clearer and "pythoniker", if you don't mind the Housism.

ceol• 14 years ago

For some reason, decorators didn't click for me until now. I've been parroting the standard decorator logic (e.g. returning a function) without knowing why.

Thanks!

simon_weber• 14 years ago

For those who prefer video, Dave Brondsema gave an excellent talk on decorators and custom context managers at PyCon: http://pyvideo.org/video/883/decorators-and-context-managers.

stock_toaster• 14 years ago

Nice description of method decorators. Didn't touch on class decorators, or decorators that can decorate both classes and methods (arguably very ugly wrapper functions that return decorators that decorate).

sudhirj• 14 years ago

Wrote a post a while ago that talks about them - http://hangar.runway7.net/decorators-wrappers-python

steamboiler• 14 years ago

Nice description. I'd suggest explaining how decorators that accept arguments (i.e. @memcached('some-arg')) work lest it befuddle some beginner. It is not straightforward (the first argument of a decorator is the function being decorated).

A primer on Python decorators

Related Stories