A quick intro guide to asynchronous programming in Python (using asyncio)

Asynchronous programming has been gaining traction amongst python’s community over the last years and it’s slowly becoming a must-known tool for any data scientist or engineer.

During the last few months, I have been using it in my own projects, while diving into myriads of articles, posts, threads, and learning the essentials. This article is what I would have loved to read when starting my journey of understanding async processes in Python.

What is asynchronous programming?

While trying to explain what asynchronism is, I found that an analogy with cooking works the best. Instinctively we cook asynchronously.

Imagine you are hungry and you want to make pasta. You don’t wanna waste a single minute and eat as soon as possible. You start by putting water in a pan. You immediately shift to cutting vegetables, you put them on a frying pan. The water boils, so you put your favorite pasta in the pan. In the meantime, you wash the cutting board, knife, table. Etc, etc.

While running your process, you don’t wanna wait a single second either. You make 3 independent requests, you don’t wanna wait until the first request was responded to make the second one. You will do the 3 requests concurrently, using asynchronism.

In Python, there are 2 ways of taking advantage of concurrency: multi-threading or asyncio. While multi-threading is a bit simpler and easier to implement, asyncio is more versatile and efficient. Thus, Im focusing on asyncio in this article.

Asyncio basic concepts

Programming asynchronously is definitely a challenge. Generally speaking, our thinking process is mostly synchronic and linear, so writing async code is somewhat counterintuitive. Thus, when building your async script, you need abstraction and complete understanding of the process you’re working with.

Asyncio is a Python library that, I believe, does a great job in bringing down to earth the complexity of concurrency. However, it does come with a few new basic concepts you need to be familiar with:

Coroutines
A coroutine is basically a function. That’s it. The main difference is that it is an awaitable object. Which basically means that you can pause and resume its run (and when paused, something else runs). The other main difference is that you can’t simply run it as a normal function. You need to use a specific asyncio syntax: asyncio.run(your_coroutine()).

Event Loop
The event loop is the bread and butter in asyncio. It’s basically the orchestra director of the coroutines, it decides what to run and when. Every time a coroutine pauses, the event loop switches the execution to another one. The beauty of asyncio is that you can manage very well without understanding how the event loop actually works.

Async/await syntax
async/await is the basic syntax you need to be familiar with. async is used to define coroutines in the following way:

Now, await is used to pause the execution of a coroutine and give the control back to the event loop. Basically, you need to await a coroutine object to do it:

Gather method
Using the gather method you can make use of concurrency very easily. In a nutshell, you input coroutines and it programs the execution for you. It also creates an event loop automatically, so every time one of your coroutines pauses execution, another will run. For example:

How to make concurrent requests using aiohttp

Say you need to make 100 requests as part of one of your processes. You won’t do them sequentially and wait for eternity, but rather concurrently. Asyncio is definitely the way to go, yet, not enough.

If you try making requests with the standard requests library, your process will actually wait for the response, no control will be given back to the event loop (because the requests methods are not coroutines). Thus, you need to use a library with async methods and that’s where aiohttp comes to the scene. Here is an example on how to use it:

Everything has cons, and so do asyncio

While researching and writing asyncio code, I did notice some cons about its use:
— Code become unreadable for people not familiar with it. Unlike threads, asyncio comes with its own syntax and tools which becomes hard to understand.
— You cant use libraries with clients not built async. For example, Google Big Query client library, does requests synchronously.
— It’s generally more complex than threading. Although, more versatile, asyncio does present new elements that adds complexity to your process.

Useful Material

In the case you want to get more familiar with asyncio, here are the key materials that helped me along the way:
— Real Python: Async IO in Python: A Complete Walkthrough
— Stack Overflow: Thread 1, Thread 2
— Youtube: Video 1, Video 2, Video 3
— aiohttp doc
— Medium: Article 1, Article 2, Article 3