The Python yield keyword explained

http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained

by interro • 13 years ago

127 49 13 years ago

stackoverflow.com

johnsonqq4• 13 years ago

It's a coroutine. Not exactly revolutionary.

Sharlin• 13 years ago

That's not exactly helpful if you don't know what coroutines are and how they work.

johnsonqq4• 13 years ago

It's precisely all the help you need to go and look up your notes on coroutines from college. You did go to college didn't you? Or are you an anti-college hipster, suddenly finding the limitations of that approach.

bthomas2• 13 years ago

I use yield and generators regularly. They just make sense, but I never knew what a coroutine is in an academic sense

johnsonqq4• 13 years ago

This is why you study CS at university before you start work as a programmer. Otherwise you end up being amazed by the basic CS concepts, or worse, you end up reinventing them badly.

Yttrill• 13 years ago

This does reflect very sadly on schools, and shows up when trying to explain things to programmers used to conventional languages, especially if you're trying to explain why C is fundamentally broken.

Roughly there are three fundamental control transfer operations: conditional jumps, subroutine calls, and coroutine calls. A subroutine is a slave, the caller is a master. With coroutines, the caller and callee are peers.

Coroutine calling is more fundamental and easier to program, but it isn't available in C.

The theory is about continuations. A continuation is just "the rest of the program". You can think of it as the program counter (PC). Subroutine calling works by passing a continuation to a routine, when the routine is finished it invokes the continuation. This is done by pushing the program counter on the stack, and the subroutine popping it with a return statement.

Coroutines work by exchange of control. When you call a coroutine you give it your current continuation to call when its ready. But also you do not call the coroutine at the beginning. You call it where it last left off: at its last continuation point.

A set of coroutines are usually called fibres. They represent interleaving of control. You can emulate them with pre-emptive threads and locks, but coroutines are synchronous and non-premptive.

Felix and Go both make heavy use of coroutines (called fthreads and goroutines). Iterators as in Python are a special case.

In general on today's badly designed CPU's you have to think of coroutines as requiring stack swapping. Threads do this which is why you can emulate coroutines with threads.

By far the most well known coroutine scheduler is .. the Operating System. Its basically a coroutine of applications. This is why you can read and write to files: the real world is event driven but callbacks are impossible to program with. So the operating control inverts the events by stack swapping so your application can be written as a master.

You think you're calling the OS, and the OS thinks its calling you. You're both masters. That's coroutines.

Python (and Felix) both have yield, but that's a special case. In general the model is reading and writing channels: yielding is just writing the "sole" channel and getting a function result is just reading it. A channel is basically a "place to swap stacks".

jtchang• 13 years ago

What a troll. The post is descriptive and helpful. Go back to whatever hole you crawled out of.

akashshah• 13 years ago

Other amazing Python answers by the same guy:

What is a metaclass in Python - http://stackoverflow.com/questions/100003/what-is-a-metaclas...

Understanding Python decorators - http://stackoverflow.com/questions/739654/understanding-pyth...

jiggy20112• 13 years ago

What I find weird about the yield keyword is that it seems to effect the execution of code that comes before the statement, causing it not to execute until later.

For example:

  def createGenerator():
     print "aaa"

  mygen = createGenerator()

outputs:

aaa

Whereas

  def createGenerator():
    print "aaa"
    yield

  mygen = createGenerator()

Outputs nothing.

gejjaxxita4• 13 years ago

The presence of the yield keyword in a function definition means the function will return a generator when it's called. Therefore "print "aaa"" is not executed on "mygen = createGenerator()" as expected. "print "aaa"" is infact executed when mygen.next() is called.

To emphasise this point, it is the existence of the yield keyword which causes the function to return a generator when it is called - it will not execute any of the code. This is why the following results in an error:

  >>> def hello(j):
  ...     if j>2:
  ...         return "big number"
  ...     elif j<2:
  ...         yield "small number"

A return keyword cannot exist within the body of a generator - it makes no sense.

A more detailed explanation can be found here http://docs.python.org/2.5/ref/yieldexpr.html

jiggy20112• 13 years ago

I think I mostly understand the idea, the presence of the yield statement transforms the nature of the parent function itself and makes evaluation lazy.

This seems a little weird syntactically though because it feels like the interpreter is somehow reading ahead in the program. It also means you don't necessarily know that this is a lazy function unless you read to the end.

A more intuitive syntax might be something like:

  def lazy createGenerator():
    print "aaa"
    yield "bbb"

andreif• 13 years ago

The answer on SO says explicitly (in bold, BTW):

> To master yield, you must understand that when you call the function, the code you have written in the function body does not run. The function only returns the generator object, this is bit tricky :-)

ars• 13 years ago

A helpful hint on the name: Yield should be read like "Yields" as in "returns" or "produces".

Not Yield as in the common multithreading command to allow another thread to run.

PS. Anyone know why they chose such a confusing name?

Someone• 13 years ago

Because it allows another thread to run. Yes, they actually are coroutines, but that is an implementation detail. Just look at it as an optimization for the case where at most one of the two threads (the one producing the values and the one consuming it) runs at a time.

kevsamuel2• 13 years ago

Hi guys, I'm the autor of the answers.

Thanks for the feedback. Because of comments like these, I decided to spend more time at training people for a living, and I'm loving it.

So don't underestimate the power of web comments, it can literally change the way people live.

emmelaich• 13 years ago

Great stuff. I do like to see the simplest possible exposition of any idea/technique though; here's my attempt at a simplest possible.

    def y():
        yield 1
	yield 2

    for i in y():
        print i

kevsamuel2• 13 years ago

You got it.

The Python yield keyword explained

Related Stories