Asynchronous iterators in python
How to write async for loops in python
Introduction:
We have all worked with iterators
and iterables
in python
by now as a developer. This post is not another difference between or comparison between iterators and iterables in python
. Let me the spare that. This post gives an overview of how asynchronous iterators
work and how to write our own asynchronous iterators
.
To get an introduction to asyncio
, please refer here.
Take away:
At the end of this post, you should be able to
- Write an asynchronous iterator.
- Understand the internals of
async
iterators. - An introduction to regular
iterators
. - How are non async iterators different from async iterators.
Now, let’s get started. Shall we?
Iterators in python:
Before discussing the async
iterators, let's quickly go over the regular iterators in python.
- An iterator must implement the
__iter__
special method. - The
__iter__
special method should return aniterable
.i.e. Any object that implements the__next__
special method. This could be its own class(self
) or any other class object. - The
__next__
method has the logic to run theiterator
until a condition is satisfied. - Once the condition is satisfied,
StopIteration
error message should be raised.
Now, let’s quickly write an iterator
, OddCounter
which iterates only through the odd numbers. I know this doesn't make more sense now. But it's fun to play around with the capability of iterators
. You could try implementing an even counter or fibonacci counter.
Output:
1
3
5
7
9
...
...
93
95
97
99
The above example was just to illustrate the working of regular iterators
in python
.Now that we have learnt it, let's jump to what we are here for.
Async iterators:
In the regular iterators, what if there is a scenario where you have to do some I/O tasks to get data inside the __next__()
method. That's exactly what async iterators
do.
An async iterator
typically contains,
- A
__aiter__()
method instead of__iter__()
method. - The
__aiter()__
method must return an object that implements aasync def __anext__()
. - The
__anext__()
method must return a value for every single iteration and raiseStopAsyncIteration
at the end instead ofStopIteration
.
Now, let’s try rewriting our OddCounter
iterator using async iterator.
Output:
1
3
5
7
9
...
...
93
95
97
99
Though this is syntactically right and works, we don’t have any compelling reason to use an async iterator
here. One such scenario would be, if we do some operation inside the __anext__()
method that requires fetching data from elsewhere.For instance, if we fetch data from a remote database or another coroutine function one at a time, then it would be the best candidate for writing asynchronous iterators.
In the above example, KeyTaker
is an asynchronous iterator that extracts keys from a coroutine
. Now you could argue this could be done with a simple class and a function. Of course we could. The point here is, all_keys()
method could be performing an I/O operation like fetching information from a database or calling an API. To keep things simple, I have just created a dictionary and returned their values for the key queried.
The main()
function just iterates through KeyTaker
and prints the values. We could also call another coroutine inside the async for
loop. That would make this code complete.
Summary:
- async iterators have __aiter__ and __anext__ special methods
- StopAsyncIteration should be raised at the end of iteration.
- async iterators could be used if we perform an I/O operation on demand where the iterator in turn is performing another I/O operation.
References:
Originally published at https://dock2learn.com on March 10, 2022.