yield and yield from

The dictionary gives two definitions for the verb "to yield": yield and concession. For yield in Python generators, both meanings are true. The yield item line of code will produce a value and provide it to the caller of next(...); in addition, it will make a concession, suspend the execution of the generator, and let the caller continue to work until it needs to use another value before calling next( ). The caller will pull the value from the generator.

From a syntactic point of view, coroutines are similar to generators in that they are functions that contain the yield keyword in their definition. However, in a coroutine, yield usually appears on the right side of the expression (for example, datum = yield), and it can produce a value or not—there is no expression after the yield keyword. The coroutine may receive data from the caller, and the caller uses the .send(datum) method to provide the data to the coroutine.

One: How does a generator evolve into a coroutine
Since the yield keyword was added to python, it has undergone a series of evolutions:
The yield keyword can be used in expressions (a = yield b);

The .send(value) method has been added to the generator API (the caller of the generator can use the .send(...) method to send data, and the sent data will become the value of the yield expression in the generator function);

PEP 342 added .throw(…) and .close() methods (the former is to let the caller throw an exception and handle it in the generator; the latter is to terminate the generator);

Therefore, the generator can be used as a coroutine. A coroutine refers to a process that cooperates with the caller to produce values ​​provided by the caller.

The recent evolution of coroutine comes from the implementation of "PEP 380—Syntax for Delegating to a Subgenerator" in Python 3.3 (https://www.python.org/dev/peps/pep-0380/）。PEP 380 made two changes to the syntax of the generator function:

The generator can return a value; previously, if a value was provided to the return statement in the generator, a SyntaxError exception would be thrown;

The new yield from syntax is introduced, which can be used to reconstruct complex generators into small nested generators, eliminating the large amount of boilerplate code required to delegate the work of generators to sub-generators.

Two: the basic behavior of the generator used as a coroutine

A coroutine can be in one of four states. The current state can be determined using the inspect.getgeneratorstate(...) function, which returns one of the following strings.

GEN_CREATED: waiting to start execution;

GEN_RUNNING: The interpreter is executing (this status can only be seen in multi-threaded applications);

GEN_SUSPENDED: pause at the yield expression;

GEN_CLOSED: End of execution;

A simple example is as follows:

>>> def simple_coro2(a):
...     print('-> Started: a =', a)
...     b = yield a
...     print('-> Received: b =', b)
...     c = yield a + b
...     print('-> Received: c =', c)
...

The output is:

>>> my_coro2 = simple_coro2(14)
>>> from inspect import getgeneratorstate
>>> getgeneratorstate(my_coro2)
'GEN_CREATED'
>>> next(my_coro2)
-> Started: a = 14
>>> getgeneratorstate(my_coro2)
'GEN_SUSPENDED'
>>> my_coro2.send(28)
-> Received: b = 28
>>> my_coro2.send(99)
-> Received: c = 99
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>> getgeneratorstate(my_coro2)
'GEN_CLOSED'

The first step of calling the next(my_coro2) function is usually called "prime" the coroutine (that is, let the coroutine execute forward to the first yield expression, ready to be used as an active coroutine).

The key point is that the coroutine suspends execution at the location of the yield keyword. In the assignment statement, the code to the right of = is executed before the assignment. Therefore, for the line of code b = yield a, the value of b will not be set until the client code activates the coroutine again.

The execution process of simple_coro2 coroutine is divided into 3 stages, as shown in the following figure:

Three: use the coroutine to calculate the moving average

The following is a coroutine for calculating the moving average:

def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield average
        total += term
        count += 1
        average = total/count

The output is:

>>> coro_avg = averager()
 >>> next(coro_avg) #call next function, pre-excitation coroutine
>>> coro_avg.send(10)
10.0
>>> coro_avg.send(30)
20.0
>>> coro_avg.send(5)
15.0

This infinite loop indicates that as long as the caller keeps sending values ​​to this coroutine, it will keep receiving values ​​and then generate results. Only when the caller calls the .close() method on the coroutine, or the coroutine is reclaimed by the garbage collector without a reference to the coroutine, the coroutine will terminate.

After calling the next(coro_avg) function, the coroutine will execute forward to the yield expression and output the initial value of the average variable—None, so it will not appear in the console. At this point, the coroutine pauses at the yield expression and waits until the caller sends the value. The coro_avg.send(10) line sends a value, activates the coroutine, assigns the sent value to term, and updates the values ​​of the three variables total, count, and average, and then starts the next iteration of the while loop to produce the average variable Wait for the next assignment of the term variable.

Four: Pre-excitation coroutine decorator

If it is not pre-excited, then the coroutine is useless. Before calling my_coro.send(x), remember to call next(my_coro). In order to simplify the use of coroutines, a pre-excitation decorator is sometimes used.

The following is an example of a pre-excitation decorator (Python3):

from functools import wraps

def coroutine(func):
    @wraps(func)
    def primer(*args,**kwargs):
        gen = func(*args,**kwargs)
        next(gen)
        return gen
    return primer    

@coroutine
def averager2():
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield average
        total += term
        count += 1
        average = total/count

The output is:

>>> coro_avg = averager()
>>> from inspect import getgeneratorstate
>>> getgeneratorstate(coro_avg)
'GEN_SUSPENDED'
>>> coro_avg.send(10)
10.0
>>> coro_avg.send(30)
20.0
>>> coro_avg.send(5)
15.0

Note that when you use yield from syntax to call the coroutine, it will automatically pre-excite

Five: termination of the coroutine and exception handling

Unhandled exceptions in the coroutine will bubble up and pass to the caller of the next function or send method (that is, the object that triggers the coroutine).

>>> from coroaverager1 import averager
>>> coro_avg = averager()
>>> coro_avg.send(40)
40.0
>>> coro_avg.send(50)
45.0
>>> coro_avg.send('spam')
Traceback (most recent call last):
...
TypeError: unsupported operand type(s) for +=: 'float' and 'str'
>>> coro_avg.send(60)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

Since no exception is handled in the coroutine, the coroutine will terminate. If you try to reactivate the coroutine, a StopIteration exception will be thrown.

Starting from Python 2.5, client code can call two methods on the generator object: throw and close, and explicitly send exceptions to the coroutine.

1：generator.throw(exc_type[, exc_value[, traceback]])

Causes the generator to throw the specified exception at the paused yield expression. If the generator handles the thrown exception, the code will execute forward to the next yield expression, and the output value will become the return value obtained by calling the generator.throw method. If the generator does not handle the thrown exception, the exception will bubble up and pass to the caller's context.

2：generator.close()

Causes the generator to throw a GeneratorExit exception at the paused yield expression. If the generator does not handle this exception, or throws a StopIteration exception (usually means running to the end), the caller will not report an error. If a GeneratorExit exception is received, the generator must not produce a value, otherwise the interpreter will throw a RuntimeError exception. Other exceptions thrown by the generator will bubble up and pass to the caller.

Examples are as follows:

class DemoException(Exception):
    """ is the exception type defined for this demonstration."""
    
def demo_exc_handling():
    print('-> coroutine started')
    while True:
        try:
            x = yield
        except DemoException:
            print('*** DemoException handled. Continuing...')
        else:
            print('-> coroutine received: {!r}'.format(x))
    raise RuntimeError('This line should never run.')

The output is:

>>> exc_coro = demo_exc_handling()
>>> next(exc_coro)
-> coroutine started
>>> exc_coro.send(11)
-> coroutine received: 11
>>> exc_coro.send(22)
-> coroutine received: 22

>>> exc_coro.throw(DemoException)
*** DemoException handled. Continuing...
>>> getgeneratorstate(exc_coro)
'GEN_SUSPENDED'

>>> exc_coro.close()
>>> from inspect import getgeneratorstate
>>> getgeneratorstate(exc_coro)
'GEN_CLOSED'

Six: Let the coroutine return value

In Python2, the return in the generator function is not allowed to return an accompanying return value. This restriction was removed in Python3, allowing coroutines to return values:

from collections import namedtuple
Result = namedtuple('Result', 'count average')

def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        term = yield
        if term is None:
            break
        total += term
        count += 1
        average = total/count
    return Result(count, average)

The output is:

>>> coro_avg = averager()
>>> next(coro_avg)
>>> coro_avg.send(10)
>>> coro_avg.send(30)
>>> coro_avg.send(6.5)
>>> coro_avg.send(None)
Traceback (most recent call last):
...
StopIteration: Result(count=3, average=15.5)

Sending None will terminate the loop, cause the coroutine to end and return the result. As always, the generator object will throw a StopIteration exception. The value attribute of the exception object holds the returned value.

Note that the value of the return expression will be secretly passed to the caller and assigned to an attribute of the StopIteration exception. This is a bit unreasonable, but it retains the normal behavior of the generator object—throwing StopIteration exception when exhausted. If you need to receive the return value, you can do this:

>>> try:
...    coro_avg.send(None)
... except StopIteration as exc:
...    result = exc.value
...
>>> result
Result(count=3, average=15.5)

Obtaining the return value of the coroutine has to go around a circle. You can use the yield from introduced in Python 3.3 to get the return value. The yield from structure will automatically catch StopIteration exceptions internally. This processing method is the same as the for loop processing StopIteration exception. For the yield from structure, the interpreter will not only catch the StopIteration exception, but also turn the value of the value attribute into the value of the yield from expression.

Seven: use yield from

Yield from is a new language structure added after Python 3.3. In other languages, similar structures use the await keyword. This name is much better, because it conveys a crucial point: when you use yield from subgen() in the generator gen, subgen will gain control and output The value of is passed to the caller of gen, that is, the caller can directly control subgen. At the same time, gen will block, waiting for subgen to terminate.

Yield from can be used to simplify the yield expression in a for loop. E.g:

>>> def gen():
... for c in 'AB':
...     yield c
... for i in range(1, 3):
...     yield i
...
>>> list(gen())
['A', 'B', 1, 2]

Can be changed to

>>> def gen():
...     yield from 'AB'
...     yield from range(1, 3)
...
>>> list(gen())
['A', 'B', 1, 2]

The first thing the yield from x expression does to the x object is to call iter(x) to get the iterator from it. Therefore, x can be any iterable object.

If the only function of the yield from structure is to replace nested for loops that produce values, this structure will most likely not be added to the Python language.

The main function of yield from is to open a two-way channel to connect the outermost caller with the innermost sub-generator, so that the two can directly send and output values, and can also directly pass in exceptions instead of being located A lot of boilerplate code for handling exceptions is added to the middle coroutine. With this structure, coroutines can delegate responsibilities in ways that were impossible before.

PEP 380 uses some technical terms used in yield from:

Delegated generator: generator function containing yield from expression;

Sub-generator: the generator partly obtained from the yield from expression;

Caller: call the client code of the delegation generator;

The following figure shows the interaction between these three:

When the delegate generator is paused at the yield from expression, the caller can directly send the data to the sub-generator, and the sub-generator sends the output value to the caller. After the sub-generator returns, the interpreter will throw a StopIteration exception and append the return value to the exception object. At this time, the delegate generator will resume.

Here is an example code for finding the average height and weight:

from collections import namedtuple

Result = namedtuple('Result', 'count average')

# Sub-generator
def averager():
    total = 0.0
    count = 0
    average = None
    while True:
        # main function sends data here 
        print("in averager, before yield")
        term = yield
        if term is None: # Termination condition
            break
        total += term
        count += 1
        average = total/count

    print("in averager, return result")
    return Result(count, average) # The returned Result will become the value of the yield from expression in the grouper function


# Delegation generator
def grouper(results, key):
     # This loop creates a new instance of averager each time, and each instance is a generator object used as a coroutine
    while True:
        print("in grouper, before yield from averager, key is ", key)
        results[key] = yield from averager()
        print("in grouper, after yield from, key is ", key)


# Caller
def main(data):
    results = {}
    for key, values in data.items():
        # group is the generator object obtained by calling the grouper function
        group = grouper(results, key)
        print("\ncreate group: ", group)
        next(group) #Pre-excitation group Coroutine.
        print("pre active group ok")
        for value in values:
            # Pass each value to the grouper The passed value finally arrives in the averager function;
            # grouper does not know what is passed in, and the grouper instance is suspended at yield from
            print("send to %r value %f now"%(group, value))
            group.send(value)
        # Pass None to the groupper, and the passed value finally reaches the averager function, causing the current instance to terminate. Then continue to create the next instance.
        # If there is no group.send(None), then the averager sub-generator will never terminate, the delegate generator will never be activated here, and result[key] will not be assigned
        print("send to %r none"%group)
        group.send(None)
    print("report result: ")
    report(results)


# Output report
def report(results):
    for key, result in sorted(results.items()):
        group, unit = key.split(';')
        print('{:2} {:5} averaging {:.2f}{}'.format(result.count, group, result.average, unit))


data = {
    'girls;kg':[40, 41, 42, 43, 44, 54],
    'girls;m': [1.5, 1.6, 1.8, 1.5, 1.45, 1.6],
    'boys;kg':[50, 51, 62, 53, 54, 54],
    'boys;m': [1.6, 1.8, 1.8, 1.7, 1.55, 1.6],
}

if __name__ == '__main__':
    main(data)

Each value sent by the grouper is processed by yield from and passed to the averager instance through the pipeline. The grouper will pause at the yield from expression, waiting for the averager instance to process the value sent by the client. After the averager instance has finished running, the returned value is bound to results[key]. The while loop will continue to create averager instances and process more values.

When the outer for loop iterates again, a new grouper instance is created and then bound to the group variable. The previous grouper instance (and the unterminated averager sub-generator instance it created) was reclaimed by the garbage collector.

The code results are as follows:

create group:  <generator object grouper at 0x7f34ce8458e0>
in grouper, before yield from averager, key is  girls;kg
in averager, before yield
pre active group ok
send to <generator object grouper at 0x7f34ce8458e0> value 40.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 41.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 42.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 43.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 44.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 54.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> none
in averager, return result
in grouper, after yield from, key is  girls;kg
in grouper, before yield from averager, key is  girls;kg
in averager, before yield

create group:  <generator object grouper at 0x7f34ce845678>
in grouper, before yield from averager, key is  girls;m
in averager, before yield
pre active group ok
send to <generator object grouper at 0x7f34ce845678> value 1.500000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845678> value 1.600000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845678> value 1.800000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845678> value 1.500000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845678> value 1.450000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845678> value 1.600000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845678> none
in averager, return result
in grouper, after yield from, key is  girls;m
in grouper, before yield from averager, key is  girls;m
in averager, before yield

create group:  <generator object grouper at 0x7f34ce845620>
in grouper, before yield from averager, key is  boys;kg
in averager, before yield
pre active group ok
send to <generator object grouper at 0x7f34ce845620> value 50.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845620> value 51.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845620> value 62.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845620> value 53.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845620> value 54.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845620> value 54.000000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce845620> none
in averager, return result
in grouper, after yield from, key is  boys;kg
in grouper, before yield from averager, key is  boys;kg
in averager, before yield

create group:  <generator object grouper at 0x7f34ce8458e0>
in grouper, before yield from averager, key is  boys;m
in averager, before yield
pre active group ok
send to <generator object grouper at 0x7f34ce8458e0> value 1.600000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 1.800000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 1.800000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 1.700000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 1.550000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> value 1.600000 now
in averager, before yield
send to <generator object grouper at 0x7f34ce8458e0> none
in averager, return result
in grouper, after yield from, key is  boys;m
in grouper, before yield from averager, key is  boys;m
in averager, before yield
report result: 
boys  averaging 54.00kg
boys  averaging 1.68m
girls averaging 44.00kg
girls averaging 1.58m

The key point that this experiment wants to show is that if the sub-generator does not terminate, the delegation generator will pause forever at the yield from expression. If this is the case, the program will not execute forward because yield from (like yield) transfers control to the client code (that is, the caller of the delegated generator).

Eight: the meaning of yield from

Using an iterator as a generator is equivalent to inline the definition of the sub-generator in the yield from expression. In addition, the sub-generator can execute the return statement to return a value, and the returned value becomes the value of the yield from expression.

PEP 380 is in the "Proposal" section (https://www.python.org/dev/peps/pep-0380/#proposal) explained in six points The behavior of yield from. The quote here is almost intact, but the ambiguous term "iterator" has been replaced with "sub-generator" and further explanation is given. The above example illustrates the following four points:

The values ​​produced by the sub-generators are directly passed to the caller of the delegated generator (that is, the client code);

The values ​​sent to the delegate generator using the send() method are passed directly to the sub-generator. If the value sent is None, then the sub-generator will be callednext() Method. If the value sent is not None, then the send() method of the sub-generator will be called. If the sub-generator throws a StopIteration exception, the delegated generator will resume operation. Any other exceptions will bubble up and pass to the delegate generator;

When the generator exits, the return expr expression in the generator (or sub-generator) will trigger the StopIteration(expr) exception to be thrown;

The value of the yield from expression is the first parameter passed to the StopIteration exception when the subgenerator terminates.

The specific semantics of yield from is difficult to understand, especially the two points of handling exceptions. The semantics of yield from is explained in PEP 380. Pseudocode (using Python syntax) is also used to demonstrate the behavior of yield from.

If you want to study that pseudo-code, it’s best to simplify it to cover only the most basic and common usage of yield from: yield from appears in the delegation generator, the client code drives the delegation generator, and the delegation generator drives Sub-generator. To simplify the logic involved, assume that the client does not call the throw(...) or close() method on the delegate generator. And suppose that the sub-generator does not throw an exception, but runs until it terminates, letting the interpreter throw a StopIteration exception. The script in the example above makes these simplified logic assumptions.

The following pseudo code is equivalent to the RESULT = yield from EXPR statement in the delegation generator (this is for the simplest case: the .throw(...) and .close() methods are not supported, and only StopIteration exceptions are handled):

_i = iter(EXPR) 
try:
    _y = next(_i)
except StopIteration as _e:
    _r = _e.value
else:
    while 1:
        _s = yield _y
    try:
        _y = _i.send(_s)
    except StopIteration as _e:
        _r = _e.value
        break
RESULT = _r

However, the reality is more complicated, because the client calls the throw(...) and close() methods, and the operations performed by these two methods must be passed to the sub-generator. In addition, sub-generators may be pure iterators and do not support throw(...) and close() methods, so the logic of the yield from structure must handle this situation. If the sub-generator implements these two methods, and inside the sub-generator, both methods will trigger exceptions. This situation must also be handled by the yield from mechanism. The caller may let the subgenerator throw an exception for no reason, and this situation must also be handled when implementing the yield from structure. Finally, for optimization, if the caller calls the next(...) function or the .send(None) method, the responsibility must be transferred and the next(...) function is called on the sub-generator; only when the value sent by the caller is not None, Only use the .send(...) method of the sub-generator.
The following pseudo code is the equivalent code of the statement: RESULT = yield from EXPR after considering the above conditions:

_i = iter(EXPR)
try:
    _y = next(_i)
except StopIteration as _e:
    _r = _e.value
else:
    while 1:
        try:
            _s = yield _y
        except GeneratorExit as _e:
            try:
                _m = _i.close
            except AttributeError:
                pass
            else:
                _m()
            raise _e
        except BaseException as _e:
            _x = sys.exc_info()
            try:
                _m = _i.throw
            except AttributeError:
                raise _e
            else:
                try:
                    _y = _m(*_x)
                except StopIteration as _e:
                    _r = _e.value
                    break
        else:
            try:
                if _s is None:
                    _y = next(_i)
                else:
                    _y = _i.send(_s)
            except StopIteration as _e:
                _r = _e.value
                break
RESULT = _r

In the pseudo code above, the exciton generator will be pre-excited. This indicates that the decorator used for automatic preview is not compatible with the yield from structure.

Reprinted in: