In this article, I would like to share an approach that you can handle concurrent API requests in Python using asyncio and aiohttp libraries.
Imagine if your requirement is to extract a large number of records and process them through API requests. Triggering API calls in one-by-one manner would take too much time especially when there are over thousand or even million requests to be processed.
Let’s take a small example, assuming that you have 10,000 requests to be processed and each request takes around 1 second in average, if you process them synchronously, the total time taken for this batch to complete will be 10,000 seconds (~2.8 hours).
However, on the same 10,000 requests, if you process it asynchronously, say 5 requests concurrently each time, it would take only 2,000 seconds (~33 minutes). That is 5 times faster than before!
In this example, I will use asyncio library which provides an ability for us to utilize async and await syntaxes that allow us to execute Python codes concurrently.
Executing code concurrently
Here is an example of how we can use asyncio library to execute codes concurrently.
The output below shows that all calculations started concurrently, while all results were returned as a list by asyncio.gather function at line 20.
Process API requests concurrently
Now let’s take an example of how we could utilize asyncio library to call API requests concurrently. Note that aiohttp library is used for handling API requests as it is:
Asynchronous HTTP Client/Server for asyncio and Python.
First, let’s find what kind of API we would use to test this code. I personally like https://mockapi.io as it provides easy-to-setup and fast mock API endpoints.
A simple GET endpoint is created. This endpoint will simply return a list of mock 50 users as shown below.
Now we have API endpoint for testing, let’s modify our code to call this endpoint 10 requests concurrently.
The output below shows that all 10 API requests started concurrently. Then the results were returned with a random response rate.
Conclusion
We have seen how we can utilize asyncio and aiohttp libraries to process multiple API requests concurrently. Now let’s say if you have a large number of API requests to be processed, you can divide them into multiple chunks, then process each chunk concurrently as the example above.
This will help save time and optimize computing resource as it does not have to be put idle waiting for a response from each request sequentially.
However, be sure to apply this to a suitable situation as it depends of a number of factors such as a workload capacity of the API server and etc.
