A crash course in Python “comprehensions” and “generators”

https://medium.com/@petefison/a-crash-course-in-python-comprehensions-and-generators-f069c8f8ca38

by rbanffy • 4 years ago

121 55 4 years ago

I have a rule for comprehensions that I try not to break: they should only have two phrases in them. Whether that’s two “for x in y” or one “for” and one “if”. Anything more becomes confusing and illegible.

Complicated comprehensions are not clever or impressive, they’re annoying.

cuteboy192• 4 years ago

Yes, the equivalent itertools expression is much more easily readable

Waterluvian4• 4 years ago

Yes!! itertools. Always be aware of itertools.

code_biologist3• 4 years ago

What do you mean by the equivalent itertools expression? It's not clear to me how an expression like this is made more readable by itertools. I'm not sure how itertools is even relevant.

    sum(
        a + b
        for a in range(10)
        if a % 2 == 0
        for b in range(10)
    )

TheAdamist• 4 years ago

Yeah i did a multi level nested one once including else clauses, and it didnt actually assign to anything, only side effects. Felt clever at the time.

But it was completely incomprehensible and unmaintainable. So got replaced by me later.

Waterluvian4• 4 years ago

omg. A comprehension for side effects only. That’s so horrible I wish I could see it. :)

code_biologist3• 4 years ago

You can abuse it, but the lineage of comprehensions (Haskell's list monad, SQL) make it clear that longer examples are certainly useful.

Haskell list comprehensions limited to two clauses, SQL queries limited to a single FROM and a WHERE, or a FROM and a JOIN sound pretty limited. I use Python comprehensions to do those same kinds of data querying without leaving the Python context, so seems weird to limit myself for the same reason.

Using imperative constructs certainly isn't wrong, but they tend to crawl off the right hand side of the page.

maxbond• 4 years ago

Yeah, I personally find a triply-nested comprehension to be more readable than a triply-nested loop. Once you get comfortable with the syntax, it's nice to be able to fit it compactly on your screen, so that your eyes can roll all over it without having to scroll. I agree with the general principle that terseness can be a detriment to readability, but sometimes it's nice for everything to be in one place.

I suppose this is only true in combination with verbose function names that capture the logic of the operation you're performing, with the comprehension expressing how this logic is composed. In my mind, comprehensions are sugar over map(), filter(), etc., and before I was using comprehensions I had a mess of nested calls to these functions with lots of lambdas. Comprehensions are a big step up from that as far as readability goes.

im3w1l2• 4 years ago

Tbh I often find I would like to use variables in sql. Something like

  americans := SELECT username FROM users WHERE country = "us";
  SELECT url, visits FROM posts, americans ON posts.author = americans.username ORDER BY visits DESC LIMIT 10;

ddejohn• 4 years ago

My rule has always been that if it doesn't fit on one line it gets re-written, or if practical, I will factor out what I can into helpers. As soon as a comprehension spills over into a new line, the readability tanks for me.

834573• 4 years ago

With proper indention, a comp with long variable names or 2 part if is very readable for me when if is on it's own line. Put from on it's own line and it is like a sql statement.

RobertoG• 4 years ago

Well, I learn a few details with the article, so, thanks.

On the other hand, and maybe is me being too literal, I think it gives the wrong impression about what a generator is. It has nothing to do with one liners using brackets. My generators use 'yield' and look like a 'for' and they are still generators.

pedrovhb• 4 years ago

The author conflates generators with generator expressions[1]. He's correct in that generator expressions lazily evaluate and lists don't, but actual generators are a much more powerful construct - they're functions which use the `yield` keyword. They're most frequently used as iterators, but they can also communicate with the outside state via receiving values (`x = yield y`), being explicitly closed with `.close()`, or having an exception raised at the yield point with `.throw()`.

They're so powerful in fact, that asyncio is implemented under the hood with a lot of generator cleverness. You rarely see it, but the `__await__` dunder method which powers `await` is just a synchronous function which returns a generator.

Generator expressions are useful for when there's memory constraints that preclude using a list comprehension, but list comprehensions are actually faster because of cPython's internal optimizations.

[0] https://peps.python.org/pep-0255/ [1] https://peps.python.org/pep-0289/

enragedcacti• 4 years ago

but they can also communicate with the outside state via receiving values (`x = yield y`)

I'm dumbfounded that I have written python for as long as I have without ever encountering this, TIL!

https://peps.python.org/pep-0342/

photochemsyn• 4 years ago

Just found an interesting example explaining how that works, I wasn't aware of it either (this site also has nice 'yield from' explainer):

https://lerner.co.il/2020/05/08/making-sense-of-generators-c...

"The above code looks a bit weird, in that “yield’ is on the right side of an assignment statement. This means that “yield” must be providing a value to the generator. Where is it getting that value from?

Answer: From the “send” method, which can be invoked in place of the “next” function. The “send” method works just like “next”, except that you can pass any Python data structure you want into the generator. And whatever you send to the generator is then assigned to “x”."

knlb2022• 4 years ago

I used this in AoC for the first time a couple of years ago, as a consequence of diving deeply into asyncio. https://explog.in/static/aoc2019/AoC23.html

tomn• 4 years ago

Back in the days before async/await, twisted (an asynchronous I/O framework) used these to emulate async/await -- you could yield a differed value (like a promise/future), and it would pass it back when the value was available, saving some effort making your state explicit and writing many callbacks:

https://twisted.org/documents/21.2.0/api/twisted.internet.de...

nijave• 4 years ago

Besides memory, they're also great for an expensive operation. You can do pagination with network calls, disk I/O, etc

henrydark• 4 years ago

Generators and generator expressions are useful every time your api involves an unbounded sequence.

I once wrote a library of signal processing components based on the premise that the raw input signal is such a sequence, and its concrete type was Iterable[np.ndarray] (iterable of chunks of audio signal). Since the signal is unbounded - we just continue sampling from, say, a microphone forever - the types can't involve finite sequences like lists. A general iterable is works here, and then components that transform the sequence are themselves implemented as generators, or if they are very simple, as generator expressions.

Consider the following function for amplifying a signal:

  def amplify(signal: Iterable[np.ndarray], a: float):
    return (a*chunk for chunk in signal)

You can flip the whole infinite streams interface and talk about "pushing" chunks around, and it's an esthetic choice.

charlieflowers• 4 years ago

To be fair, I don’t see a conflation here. He chose not to elaborate on generator functions, but that appears to be due to a defensible conscious choice about article scope.

Imo it’s a well written, technically accurate article.

buttocks• 4 years ago

I love explainers like these because my Python code looks like C without the curly braces and semicolons.

henron• 4 years ago

I really prefer working with functional transformations over containers (including iterators).

saeranv2• 4 years ago

What's a functional transformation?

ghostwriter• 4 years ago

fmap, fold, traverse, join

ggm2• 4 years ago

A parenthetical comment: I've always liked the imagery a "crash" course gives me:

Either its the course of treatment you get after a crash

Or it's the course of actions which is going to lead to a crash.

eyelidlessness• 4 years ago

I’ve always taken it to mean it’s fast and unthorough, which I suppose aligns with your latter case but I think it’s a specialized usage still.

ggm2• 4 years ago

It's usually the least worst choice for somebody thrown into unexpected situations. Deliberately walking to a crash course on anything to avoid the long learning path is like autodidactism: it works, kinda, and kinda not.

I should add I'm a serial offender.

js22• 4 years ago

I find chain much easier to read than nested comprehensions:

    from itertools import chain
    list(chain(*trees))
    list(chain.from_iterable(trees))
    list(chain(*dog_breeds.values()))
    list(chain.from_iterable(dog_breeds.values()))

Jensson• 4 years ago

Nested comprehensions are a more powerful construct, you can do more with it than chain or the itertools module. Chain is cleaner for this case since it is weaker, so it is clearer what it does, but since list comprehension is more powerful you ought to learn it first and chain is just a nicety.

js22• 4 years ago

I’ve been using Python as my primary language since before it had list comprehensions and I still find nested comprehensions to be hard to comprehend, hard to compose, and hard to refactor.

I usually resort to building up generator expressions, using itertools, or writing traditional loops.

$0.02.

antirez• 4 years ago

I always thought list comprehension shows the problem with Python: too many ad hoc ideas. It's a language that looks like a lot more like a sum of many tools than a few well chosen orthogonal ideas. Ruby is a much better language with a much worse implementation, community, set of libraries.

aunty_helen• 4 years ago

Nice article. I learnt some stuff.

WRT to length of a statement, look at how many operations are being done. If it’s doing something like filtering that should be 2-3 ops on the same line. Perfect application of a comprehension with an if statement.

If it’s 5+ different things jammed into a comprehension, you’re failing PR. Don’t think it needs to hit 2 LOC before refactoring.

Build a list, take a property, call a function with that value, iterate, if something, else something, all on one line?? Aren’t you clever. Now, let’s write professional code.

gompertz• 4 years ago

For those interested... Generators first appeared in Icon programming language in 1977. https://en.m.wikipedia.org/wiki/Icon_%28programming_language... (ctrl+f Generators and Goal Directed evaluation)

Incredible it has taken 40+ years for some languages to just catch up to the concept!

antod• 4 years ago

40yrs? Python has had generators for approx 20yrs already. Unless you're talking about a different language.

xwowsersx2• 4 years ago

   In a REPL session, "_" means "the previous output"

Wow TIL. After all these years. I would always just arrow up and go to the beginning of the line to bind to a variable if I missed it the first time. Nice to learn something new, however small!

JonathanMerklin2• 4 years ago

I just checked and this is true for the Node.js REPL (Tried in v12, v14, v16, and v18) as well, which I'm also just learning for the first time (and I had the same workflow as you before this). Neat!

xwowsersx2• 4 years ago

Ha! Funny that you'd even think to check. In Scala and Python (I'm guessing other langs as well), the underscore can be used for ignored variables, among other things. Never thought it had another usage in the REPL. Actually really useful!

pizza• 4 years ago

Neat trick with generators: refactoring out related looping structures across multiple different functions, for greater reusability

A crash course in Python “comprehensions” and “generators”

Related Stories