A Quick Guide To Python Generators and Yield Statements

First, a quick review of iterables and list comprehensions.

Iterables

An iterable is an object that can be iterated over. In other words, an object that can be used in a forloop. A Python list is the most common example.

Code

data = [0, 1, 2, 3, 4]
for each in data:
    print(each)

Output

0
1
2
3
4

List Comprehensions

List comprehensions provide a concise way to create lists from other lists.

Code

data = [0, 1, 2, 3, 4]
result = [x*x for x in data]
for each in result:
    print(each)

Output

0
1
4
9
16

Generators

Generators are functions that allow you to declare a function and have it behave like an iterator. Going forward, it will be important to remember that functions are first class objects.

Code

data = [0, 1, 2, 3, 4]
new_generator = (x*x for x in range(5))
for each in new_generator:
    print(each)

The code above is a very simple generator. It is very similar to the list comprehension from earlier.

Output

0
1
4
9
16

Unlike a list, a generator only can be used once. When it is empty. It is empty.

Code

data = [0, 1, 2, 3, 4]
new_generator = (x*x for x in range(5))
for each in new_generator:
    print(each)
for each in new_generator:
    print(each)

The code above is a very simple generator. It is very similar to the list comprehension from earlier.

Output

0
1
4
9
16

There is nothing left to iterate.

Yield

yield is a keyword just like return.

Code

def generatorGenerator():
    for x in range(5):
        yield x*x
new_generator = generatorGenerator()
for each in new_generator:
    print(each)

Each time we iterate over new_generator , we step through the function. In this example the yield is inside a for loop.

Output

0
1
4
9
16

Here is another example

Code

def multiYield():
    yield "Hello"
    yield "World"
    yield "!"
for each in multiYield():
    print(each)

Once again, each time we iterate over multiYield , we step through the function. Each time me move through the code.

Output

Hello
World
!

And one more

Code

def multiYield():
    x = 5
    yield x
    x = x + 5
    yield x
    x = x * x
    yield x
for each in multiYield():
    print(each)

In this example we can see that the function’s internals are not static. The value of x changes with each yield.

Output

5
10
100

What are they good for?

Generators are lazy. They only work on demand. That mean they can save cpu, memory, and other resources. Take a look at the following:

Code

import sys
result = [x*x for x in range(1000000)]
memory_size = sys.getsizeof(result)
print(memory_size)

We have to calculating the square of 1,000,000 numbers and store them in memory.

Output

8697464

But what happens when it is 1,000,000^n numbers? Eventually we will run out of resources. It will also be very slow. Or we can use generators that will do it one at a time.

Fibonacci

Every Python tutorial about yield and generators must include a Fibonacci sequence. I am pretty sure this is a law. I am not sure who enforces it. But here is my Fibonacci.

Code

def fibonacci():
    a = 0
    b = 1
    while True:
        yield a
        old_a = a
        a = b
        b = old_a + b
for each in fibonacci():
    print(each)
    if each > 20:
        break

Without `break` loop would be endless.

Output

0
1
1
2
3
5
8
13
21

I hope you found this helpful. If you want more, I have a channel full of Python and Linux video tutorials on Youtube.