A collection of not so obvious Python stuff

http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/not_so_obvious_python_stuff.ipynb

by adamnemecek • 12 years ago

88 15 12 years ago

nbviewer.ipython.org

What I love about Python is that these hangups are mostly relatively obscure, due to the puritanical adherence to principle of least surprise by Guido and the other language maintainers. No language is completely without quirks though, especially one as old as Python.

jsmeaton• 12 years ago

This bit bothered me slightly:

But what if we put a mutable object into the immutable tuple? Well, modification works, but we also get a TypeError at the same time.

    tup = ([],)
    tup[0] += [1]

Of course that's an error. It's equivalent to:

    temp = tup[0]
    temp += [1]
    tup[0] = temp

But the given explanation is way too obtuse. This shouldn't be a mystery at all:

If we try to extend the list via += "then the statement executes STORE_SUBSCR, which calls the C function PyObject_SetItem, which checks if the object supports item assignment. In our case the object is a tuple, so PyObject_SetItem throws the TypeError. Mystery solved."

rix0r• 12 years ago

It's not ENTIRELY obvious. For example, tup[0].extend([1]) would work fine, and the behaviour of __iadd__() seems like it should be the same as extend(). In fact, the reference[1] has the following thing to say:

For instance, to execute the statement x += y, where x is an instance of a class that has an __iadd__() method, x.__iadd__(y) is called.

However, this is patently not true, apparently:

    tup = ([],)
    tup[0].__iadd__([1])  # Works
    print tup

    tup = ([],)
    tup[0] += [1]  # TypeError
    print tup

[1] https://docs.python.org/2/reference/datamodel.html#emulating...

emehrkay• 12 years ago

I somehow didn't know about nonlocal. With that, isn't python 100% comparable with JavaScript?

abecedarius• 12 years ago

http://ballingt.com/2014/03/17/python-javascript.html makes an extended comparison.

Grue32• 12 years ago

Python didn't even have a nonlocal statement until 2.7. Making closures was (and still is, since I have to work with 2.6) a total nightmare.

random_coder• 12 years ago

nonlocal is only in python 3.

zwegner2• 12 years ago

The "consuming generator" problem is a good point, but their workaround misses a better solution. In Python 3, range() returns a range object, which has a __contains__ method, so you can use "in" just fine without consuming any generators. It's also much faster (it would be constant time/memory except for the fact that integers can have arbitrary size).

    l = range(5)
    print('2 in l,', 2 in l)
    print('3 in l,', 3 in l)
    print('1 in l,', 1 in l)

prints...

    2 in l, True
    3 in l, True
    1 in l, True

Also, the lambda/closure example is something that really irritates me about Python. Both dynamic and lexical scoping have this problem: they always reevaluate the variable inside the closure whenever it's called, the only difference is which parent scope the value is pulled from (the parent scope or the calling scope). It's not that the "last lambda is being reused", as the article says, but rather, there are five different lambdas, but they each refer to the same variable, which is set to 4 after the list comprehension. This sucks for metaprogramming, where you'd like to create a bunch of copies of a function with different parameterizations. There's two workarounds that I know of for this for dynamic scoping: creating another nested scope, or using keyword arguments. I'm not sure what you'd do for lexical scoping (but who uses that anyways?).

    for x in [(lambda i: lambda: i)(i) for i in range(5)]:
        print(x())

...or...

    for x in [lambda i=i: i for i in range(5)]:
        print(x())

In my purely-functional Python-like programming language, Mutagen, closures are static, so that they capture the value of variables from the enclosing scope at the point that they're defined. So this works as you'd expect, printing 0 and 1 (no list comprehensions yet, and lambdas are full functions, and of course lists are immutable so no +=):

    lambdas = []
    for x in [0, 1]:
        lambdas = lambdas + [lambda: return x;]
    for l in lambdas:
        print(l())

zwegner2• 12 years ago

Oops, just realized I mixed up dynamic/lexical scoping, and it's too late to edit. Python uses lexical scoping.

emidln• 12 years ago

The first workaround is actually the same as what you do for a map in 2.x (that always works):

    for x in map(lambda i: lambda: i, range(5)):
        print(x())

andolanra• 12 years ago

I'm being a bit pedantic, but the last snippet under About lambda and closures-in-a-loop pitfall is arguably misleading when it says, "This, however, does not apply to generators…" Which is true in the example given, but not true in general. Generators don't actually provide a new environment on each evaluation, they just lazily construct their contents, which assuages the problem in certain situations. This snippet

    for l in (lambda: n for n in range(3)):
        print(l()) # prints 0, 1, 2

will alternately request an argument from the generator, return a lambda, call it, and only then request the next argument from the generator, which means at the time of the funtion call, n will not have been incremented up to its final value. We can see the previous closures-in-a-loop behavior return if we keep the closures around to execute again, at which point the loop environment contains the final value of n:

    funcs = []
    for x in (lambda: n for n in range(3)):
        funcs.append(x)
        print(x()) # prints 0, 1, 2
    for y in funcs:
        print(y()) # prints 2, 2, 2

So generators don't 'fix' the problem by constructing distinct environments for each loop iteration, but by deferring evaluation of their body until necessary.

MereInterest• 12 years ago

Yeah, that part stood out to me as well. The only way I've found to get different values for each of the functions is to use a different closure for each functions.

    funcs = [(lambda i: (lambda :i))(n) for n in range(3)]
    for f in funcs:
        print(f())

d64f396930663ee• 12 years ago

Even after reading this list, I'm comfortable forgetting everything (with maybe one exception) because these surprises really only come up when you use horrible coding practices. But it's nice to know it really takes some serious effort to find these kinds of things in Python.

allendoerfer• 12 years ago

Midnight is not False in Python 3.5 anymore:

https://docs.python.org/3.5/whatsnew/3.5.html

A collection of not so obvious Python stuff

Related Stories