Python Building Blocks - Iterators

Iterators are at the core of Python. Just like when we were discussing Generators here, you have already worked with iterators if you have worked with python for more than an hour. Knowing about iterators and how they function can help you make the best possible use out of them.

In this post, we will discuss iterators and what it takes for something to be called an iterator. We will also look at how iterators are different from iterables. Yes, there is a subtle difference between these two.

I think there’s an unspoken rule in the python ecosystem that whenever someone is talking about iterators, iterables, generators, containers etc, they are supposed to start with the most confusing statement that there is. I’ll not break that rule, so here it is…

An Iterable object gives an iterator which acts as a value generator. An iterable by itself is a container which hold the value.

Ok, with that out of the way, let’s move on to breaking down these terms and understanding how do we make use of them.

A container is nothing but a data structure that can hold elements or values. Lists, dicts, tuples or even strings are examples of containers.
An iterable is an object that can return an iterator. It holds the value or the items itself, the reference to the values or the logic to generate these values. The first thing that becomes evident is that - all containers can be treated as iterables. It may not be true the other way round though. For eg, a database pointer can be treated as an iterable but it doesn’t have any values contained within it but only the logic to get them from the database when requested (generally speaking wrt the best practices). It’s important to note that iterables can also have infinite values with only the logic to get to the next value defined. Eg. An iterable for the fibonacci series can return an infinite number of values.
There are other details about them such as the fact that containers should also support tests for membership etc. This is just a brief introduction on these topics. We will look into them in detail in their dedicated posts.

Finally, let’s come to the iterators. As we saw, an iterable will give an iterator. One can then use this iterator to generate the values and perform operations on them. Notice how I’m saying, iterators are used to “generate” the value and not simply “get” the values. As we saw earlier, iterators may not know about the values in advance. All they are supposed to know about is to return the next value when asked for it.

Let’s start by looking at a simple example:

As you can see, we can take a list and convert it into an iterator by using iter method. This will give an object with the data type of list_iterator . From there on, you can use the next method to get to the values of the iterator. This example highlights a few key aspects of iterators and what makes an object an iterator in the first place:

1. An iterator object should support, at minimum, the methods - __iter__ to get an iterator object as well as __next__ to get the next item or value. These are collectively known as the iterator protocol (docs).
2. An iterator can be a finite one which throws the StopIteration exception when it reaches the end of its cycle. One cannot restart the loop or reset the counter i.e. it moved in one and one one direction.

With that, we can easily build our own iterator. We need a class that follows the iterator protocol and only moves ahead. Let’s build an iterator for the Fibonacci series. To start with we will build a finite iterator.

Now let’s look at how you could consume this iterator…

As you can see, given that we have limited the iterator to only give series till 20, it will throw an exception after giving 13 and the next number in series will be 21.

You may not have recognised this access method though. The for — in loop might be more familiar to you. You can simply use this iterator in a for loop as well…

Finally, you may have noticed that it is fairly straightforward to update this iterator such that it generates infinite values in the Fibonacci series. Can you do that ?

Iterators -vs- Generators:

You may have noticed, generators and iterators seem to have overlapping features. They both seem to do the same job of generating values at run time, when it is needed rather than precomputing and maintaining them.

A generator is a specialised form of iterator.

Generators are easy to create iterator that satisfy the same conditions. We can see that in the following example.

So, why Iterators ?

Now that we have seen what iterators are and how to use them, it’s important to also understand when to use them.

Iterators are extremely useful when you are designing for memory sensitive applications or for use cases wherein, you may have to stop the iteration of an object as soon as a condition is met.
It may not come as a surprise but every use case wherein you found generators to be a good fit, iterators are a good fit as well. In the previous article on generators available here, we go over these patterns in detail.

Iterators can also make the code clean and increase its readability. They become a more natural addition to your program with the addition of few methods or access patterns. You might already be structuring your code in Classes and they can be converted into an iterator easily.
This addition makes the code easy to understand. Remember that in iterators there’s an explicit need of having a method that has the logic to get the next element. It cannot get more direct than that.

Conclusion

Iterators are very powerful when used correctly. It solves two important problems in one go - it makes the data flow in the code easy to understand thus improving the readability and it makes it easy to optimise the code for memory.

Do share a few cases in which you have used iterators and how it has helped you.