Use HTTPX for async requests

HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.

How to Install HTTPX

You can install it like a regular python package using pip

Install HTTPX using pip:

pip install httpx

Or you can install using command line by

# The command line client is an optional dependency. 
$ pip install 'httpx[cli]'

The command line installation helps you to use HTTPX directly from the command-line.

Usage

Features

HTTPX builds on the well-established usability of requests, and gives you:

• A broadly requests-compatible API.
• Standard synchronous interface, but with async support if you need it.
• HTTP/1.1 and HTTP/2 support.
• Ability to make requests directly to WSGI applications or ASGI applications.
• Strict timeouts everywhere.
• Fully type annotated.
• 100% test coverage.

To know about additional features refer https://www.python-httpx.org/

Dependencies

The HTTPX project relies on these excellent libraries:

• httpcore - The underlying transport implementation for httpx.
• h11 - HTTP/1.1 support.
• certifi - SSL certificates.
• rfc3986 - URL parsing & normalization.
• idna - Internationalized domain name support.
• sniffio - Async library autodetection.

The httpx also supports the methods regularly provided by requests such as post, get .

But the main advantage comes with async operation

What is non-blocking code?

You may hear terms like “asynchronous”, “non-blocking” or “concurrent” and be a little confused as to what they all mean. According to this much more detailed tutorial, two of the primary properties are:

Asynchronous routines are able to “pause” while waiting on their ultimate result to let other routines run in the meantime.

Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

So asynchronous code is code that can hang while waiting for a result, in order to let other code run in the meantime. It doesn’t “block” other code from running so we can call it “non-blocking” code.

The asyncio library provides a variety of tools for Python developers to do this, and aiohttp provides an even more specific functionality for HTTP requests. HTTP requests are a classic example of something that is well-suited to asynchronicity because they involve waiting for a response from a server, during which time it would be convenient and efficient to have other code running.

Lets look at an example to understand how to write asynchronous requests.

import asyncio
import httpx


async def main():
    cat_url = 'https://api.thecatapi.com/v1/images/search'

    async with httpx.AsyncClient() as client:

        resp = await client.get(cat_url)

        cat = resp.json()
        print(cat['url'])

asyncio.run(main())

Here in this code we’re creating a coroutine called main, which we are running with the asyncio event loop. Here, we are making a request to the Cat API and then awaiting a response.

This async keyword basically tells the Python interpreter that the coroutine we're defining should be run asynchronously with an event loop. The await keyword passes control back to the event loop, suspending the execution of the surrounding coroutine and letting the event loop run other things until the result that is being "awaited" is returned.

Now that we got a basic idea let’s try to understand the sample use case

Making a large number of requests

Making a single asynchronous HTTP request is great because we can let the event loop work on other tasks instead of blocking the entire thread while waiting for a response. This functionality truly shines when trying to make a larger number of requests. Let’s demonstrate this by performing the api call to

Let’s take the previous request code and put it in a loop, updating which Chuck Noris joke data is being requested and using await for each request:

Comparing speed with synchronous requests

Let’s try to run the same program but in synchronous mode

I have used requests for synchronous fetching.

As you can see joke.py which runs asynchronously took 24s and joke_sync.py took almost double the time.

Concluding Thoughts

As you can see, HTTPX is one another library available in python for performing async and sync requests, Is it better than requests that’s a one another for discussion.

If you’re interested in this library check out https://www.python-httpx.org/