Python — Iteration Protocol — Part 2
In the previous article, we learned to implement sequence and iteration protocols. In this article, we will learn about generators and implement the iteration protocol using generates. We will conclude this article by learning about itertools, a Python module that provides various functions to create and manage iterators.
Generators
A generator is like a regular Python function. It has a yield statement instead of a return statement. This differentiates a regular function from a generator.
A generator function pauses the execution of a function when it executes a yield statement. A subsequent call to the function resumes the execution of the function. Python maintains the state of a generator function when it yields. A yield statement sends a value back to a caller function.
Note: a generator function can have multiple yield statements.
When a function has a yield statement, Python internally makes it an iterator and implicitly implements the __iter__ and __next__ functions. Like an iterator, it raises an exception StopIteration when a generator terminates.
A generator provides a “syntactic sugar” for writing an iterator.
A generator helps with less memory usage and better performance by executing the code as and when needed, a concept called lazy execution.
We can create a generator either through a generator function or generator expression.
Implementing Generators
Generator function
This is a simple implementation of a generator function with 3 yield statements.
Generator expression
Following is a generator expression implementation. This is very similar to list comprehension. Instead of using square brackets [], we use parentheses ().
Note: A generator expression is used for iteration. Unlike a list comprehension, which creates a list, the generator expression does not create a list. Second, we cannot reuse a generator expression.
Generator function for user-defined types
In this section, we will see the iterable iterator and iterator implementation using the generator function.
We will continue with the same example of managing a book collection. For this, we have a class called “MyBookCollection.” This class will maintain a list of books using a linked list and implement a function called add_book to add books to the collection.
Each book is represented by a class Book.
Iterable Iterator
For a class to be an iterable iterator, it must implement 2 functions, __iter__ and __next__. Since we will implement this functionality using a generator function, a generator function (function with a yield statement) gets an implicit implementation of both __iter__ and __next__. Since the class provides the functionality of an iterable, it implements its own __iter__ function. This __iter__ function returns the generator object of the generator function implemented in a class.
We do not need to implement the __next__ function in the class as it is now implicitly available with the generator function. Refer to the code snippet below.
Iterator protocol
In this case, the iteration functionality for BookCollection is implemented as a separate class. We will implement this in the class BooksIterator.
The class BookCollection will implement its own __iter__ function. This function will create a new object of a BooksIterator class and return the generator object to the caller. While creating a new object of BooksIterator, it will pass the reference of the books.
The class BooksIterator implements a generator function that iterates through the books. It receives the books’ references from the class BooksCollection. The implementation is as follows.
Generator chaining
As the name suggests, it is a technique for creating efficient data processing pipelines. In this, multiple iterators are chained together. It is a concept for applying a sequence of operations to data. In the simple example, we write three generator functions. The first function generates a sequence of numbers. The second generator function squares the given sequence of numbers. The third generator function adds 100 to a given sequence of numbers.
We put these in a pipeline where the numbers generated by the first generator function give an input for the second generator function, and finally, the output of the second generator function becomes the output of the third generator function.
Module Itertools
As we have discussed iterables and iterators, the itertools module deserves a space here.
This module provides the functionality of iterating through the data in a memory-efficient way. It also writes easily understandable and maintainable code. We can use this module to iterate over iterables and create complex operators. A few of the examples are…
Infinite Iterators
count(start=0, step=1)
This function returns evenly spaced values that begin with the start. It accepts floating point values as well.
Cycle(iterable)
This function returns elements from the given iterable. While doing so, it maintains a copy of the iterable. All the subsequent returns of elements are made from the copy of the iterable. Be careful of the length of the iterable passed to this function.
repeat(object[, times])
This function returns an object a given number of times. If the times is not mentioned, the object is returned infinitely.
Combinatoric Iterators
product(*iterables, repeat=1)
The function returns the cartesian product of the given iterables. And the repeat repeats the list of iterable(s)
permutations(iterable, r=None)
This function returns all possible orderings as a tuple of a given length r. When r is not mentioned, it is the length of the iterable. There is no repeating element.
combinations(iterable, r)
This function returns all possible tuples of a given length r with no repeat elements.
Apart from the above, the module offers many more functions that pique interest, such as accumulate, starmap, groupby, compress, chain, etc.
Summary
- Generators are special type functions that implement iteration protocol.
- When called, the generator function returns a generator object and does not start executing a function.
- The yield statement allows us to suspend the execution of a function and pass value from the function.
- Generator expressions are similar to list comprehension. However, they do not create a list object, and it is not possible to reuse a generator expression.
- Generator functions can be chained to form a data processing pipeline.
References
Python Documentation
6. Expressions
6.2.9. Yield expressions
6.2.9.1. Generator-iterator methods
itertools — Functions creating iterators for efficient looping
Books
Python Tricks: A buffet of awesome Python features.
Effective Python: 59 Specific Ways to Write Better Python
Did you enjoy this article? Please clap it and share it with your friends.
If you have found this article helpful, please subscribe to read more articles.
Feel free to comment!!
