Asynchronous iterators in python

How to write async for loops in python

https://dock2learn.com/tech/asynchronous-iterators-in-python/

Introduction:

We have all worked with iterators and iterables in python by now as a developer. This post is not another difference between or comparison between iterators and iterables in python. Let me the spare that. This post gives an overview of how asynchronous iterators work and how to write our own asynchronous iterators.

To get an introduction to asyncio, please refer here.

Take away:

At the end of this post, you should be able to

• Write an asynchronous iterator.
• Understand the internals of async iterators.
• An introduction to regular iterators.
• How are non async iterators different from async iterators.

Now, let’s get started. Shall we?

Iterators in python:

Before discussing the async iterators, let's quickly go over the regular iterators in python.

• An iterator must implement the __iter__ special method.
• The __iter__ special method should return an iterable.i.e. Any object that implements the __next__ special method. This could be its own class(self) or any other class object.
• The __next__ method has the logic to run the iterator until a condition is satisfied.
• Once the condition is satisfied, StopIteration error message should be raised.

Now, let’s quickly write an iterator, OddCounter which iterates only through the odd numbers. I know this doesn't make more sense now. But it's fun to play around with the capability of iterators. You could try implementing an even counter or fibonacci counter.

Output:

1
3
5
7
9
...
...
93
95
97
99

The above example was just to illustrate the working of regular iterators in python.Now that we have learnt it, let's jump to what we are here for.

Async iterators:

In the regular iterators, what if there is a scenario where you have to do some I/O tasks to get data inside the __next__() method. That's exactly what async iterators do.

An async iterator typically contains,

• A __aiter__() method instead of __iter__() method.
• The __aiter()__ method must return an object that implements a async def __anext__().
• The __anext__() method must return a value for every single iteration and raise StopAsyncIteration at the end instead of StopIteration.

Now, let’s try rewriting our OddCounter iterator using async iterator.

Output:

1
3
5
7
9
...
...
93
95
97
99

Though this is syntactically right and works, we don’t have any compelling reason to use an async iterator here. One such scenario would be, if we do some operation inside the __anext__() method that requires fetching data from elsewhere.For instance, if we fetch data from a remote database or another coroutine function one at a time, then it would be the best candidate for writing asynchronous iterators.

In the above example, KeyTaker is an asynchronous iterator that extracts keys from a coroutine. Now you could argue this could be done with a simple class and a function. Of course we could. The point here is, all_keys() method could be performing an I/O operation like fetching information from a database or calling an API. To keep things simple, I have just created a dictionary and returned their values for the key queried.

The main() function just iterates through KeyTaker and prints the values. We could also call another coroutine inside the async for loop. That would make this code complete.

Summary:

• async iterators have __aiter__ and __anext__ special methods
• StopAsyncIteration should be raised at the end of iteration.
• async iterators could be used if we perform an I/O operation on demand where the iterator in turn is performing another I/O operation.

References:

https://peps.python.org/pep-0525/

Originally published at https://dock2learn.com on March 10, 2022.