What does requests offer over urllib3 in 2022?

Slightly blurry photo of a screen containing Python code that runs a request with urllib3

The Python HTTP library urllib3 has begun its version 2 journey over the past few weeks, with a couple of alpha releases being made available. This library is an absolute cornerstone of the Python ecosystem, and is a dependency of — among other things — pip, Python’s ubiquitous package management tool. But urllib3 is also the major dependency of another, arguably even more famous, Python package — the other popular HTTP library, requests.

Requests is the go-to HTTP library for many Python users, and even many people not embedded in the Python community will have heard of it. It brands itself as “HTTP for Humans™”, claiming elegance and simplicity. But do these claims hold up? Is it a well-honed, full-featured, easy-to-use tool for developers needing to work with HTTP, or is it nothing more than a wrapper around urllib3?

Let’s find out.

My aim in this article is to compare the latest versions of both urllib3 and requests in terms of usability and simplicity, through a few common use cases. At the time of writing, requests is on version 2.28.1, and urllib3 has just made 2.0.0a2 available. It is these versions specifically that I have compared.

A simple GET request

The obvious starting point for a comparison is a basic HTTP request. And a good test server to use for this is the one at httpbin.org. This server was incidentally also built by the author of requests.

We begin by comparing the code required to carry out a simple HTTP GET with each library. As with all examples on this page, we demo requests first, and then urllib3:

>>> from requests import get
>>> r = get('https://httpbin.org/get')
>>> r.status_code
200

>>> from urllib3 import request
>>> r = request("GET", 'https://httpbin.org/get')
>>> r.status
200

So there’s not a lot of difference, just a few small details. Both examples require a single import, both have the same number of lines of code, and both are just as easy to read.

The main difference between the two is how the HTTP method itself is encoded. In urllib3, "GET" is passed as an argument; with requests, this detail has been promoted to the method name, get. Both approaches require the user to know the term “get”, so this is largely just a subjective API design choice.

The requests library also offers a general-purpose request function, but this is considered by the docs to be “advanced”. Each library offers several ways to make requests, but leans towards a different recommendation by default. From the start, requests suggests that developers use the global methods get, post, etc, which persist no explicit state between successive usages. The Session object can be used for cases where persistence is required, but documentation for this is also hidden away as “advanced” usage.

On the other hand, urllib3 opens its user guide by mentioning the PoolManager This is its (slightly awkwardly-named) carrier of state between successive HTTP calls. The global request method used in the examples here is mentioned as an aside for scripting and simpler use cases. The difference in how each approach is documented by the two projects suggests a slight difference in target audience.

Conclusion: there’s very little to choose between the two libraries for simple HTTP requests, beyond small subjective differences.

Mixing in some auth

Moving on to something a little more advanced, let’s look at passing auth credentials in a GET request. Firstly, we’ll set up a few new variables:

>>> user = 'user'
>>> pswd = 'pass'
>>> uri = f'https://httpbin.org/basic-auth/{user}/{pswd}'

Using these, the code required for each library is as follows:

>>> r = get(uri, auth=(user, pswd))
>>> r.status_code
200

>>> from urllib3 import make_headers
>>> r = request("GET", uri, headers=make_headers(basic_auth=f'{user}:{pswd}'))
>>> r.status
200

While neither example is overtly complicated, requests is slightly more succinct here. And urllib3 even uses an extra import. Why?

The requests API has been simplified by tightly coupling the construction of the Authorization header into the get function. An extra import isn’t required, as this is all dealt with inside the function. But this tight coupling is a trade-off. The auth argument is useful so long as the user is using basic HTTP authentication, but it can’t be used for every type of auth. Bearer tokens are one common example for which this wouldn’t apply.

Urllib3 takes a more explicit and decoupled approach. As a result, a similar code pattern can be applied for any kind of auth. The make_headers helper is actually completely optional, and in practice just provides similar syntactic sugar to requests’ auth argument, albeit with a few more words.

Let’s look at a another example, this time for the aforementioned bearer tokens:

>>> token = "abcd1234"
>>> uri = "https://httpbin.org/bearer"

>>> r = get(uri, headers={"Authorization": f"Bearer {token}"})
>>> r.status_code
200

>>> r = request("GET", uri, headers={"Authorization": f"Bearer {token}"})
>>> r.status
200

Now the differences have melted away. Both examples use identical headers arguments, and both require the user to know the header formats. From my reading of the docs, neither library provides any helpers or syntactic sugar for this particular use case.

Conclusion: while both libraries provide a little extra support for HTTP basic auth, requests can also save a few additional keystrokes; no help is provided by either library for other kinds of auth, however.

Reading the responses

Up until now, we’ve only looked at the request side of the HTTP exchange. So let’s move on to the responses. For this, I’ve chosen to explore three types of content: image/png, application/json, and text/plain.

The first of these, image/png, deals with binary data. We can explore the type and content of the response with each of the two libraries as follows:

>>> r = get("https://httpbin.org/image/png")
>>> r.headers['content-type']
'image/png'
>>> r.content[:16]
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR'

>>> r = request("GET", "https://httpbin.org/image/png")
>>> r.headers['content-type']
'image/png'
>>> r.data[:16]
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR'

And as with many of our other examples, there’s no major difference here, aside from a bit of naming. The number of lines of code is the same, and neither is any harder or easier to read.

So let’s move swiftly on to application/json:

>>> r = get("https://httpbin.org/ip")
>>> r.headers['content-type']
'application/json'
>>> r.json()
{'origin': '123.45.67.89'}

>>> r = request("GET", "https://httpbin.org/ip")
>>> r.headers['content-type']
'application/json'
>>> r.json()
{'origin': '123.45.67.89'}

And yet again, the examples are almost identical. But it’s worth pointing out that the .json() method has only been added to urllib3 in its version 2 pre-releases. This isn’t available if you’re using an earlier version, whereas requests has supported this for some time.

Finally, let’s explore the text/plain use case:

>>> r = get("https://httpbin.org/robots.txt")
>>> r.headers['content-type']
'text/plain'
>>> r.text
'User-agent: *\nDisallow: /deny\n'

>>> r = request("GET", "https://httpbin.org/robots.txt")
>>> r.headers['content-type']
'text/plain'
>>> r.data.decode("ascii")
'User-agent: *\nDisallow: /deny\n'

At first glance, it may not seem like there’s much difference here either. But requests is actually providing some additional syntactic sugar for retrieving textual content: the .text accessor.

Between computer systems, text content is typically transferred according to a particular character encoding, such as ASCII, ISO-8859–1, or UTF-8. This encoding provides the rules by which raw bytes can be converted to and from character data. Over HTTP, the encoding can be included as a parameter, alongside the Content-Type header. For example:

Content-Type: text/plain; charset=utf-8

If the charset parameter is not supplied, RFC 2046 states that text data should be interpreted as ASCII. Our urllib3 example above injects that knowledge directly — we’ve assumed here that the developer knows how to decode the raw response data into text, or has obtained that information out-of-band.

Requests takes a slightly different approach — you’ll see that there’s no explicit mention of ASCII in the example. Instead, what’s happening is that requests is internally deferring to the (very clever) chardet library to guess the correct encoding. Something which a urllib3 user could also choose to do by including an explicit dependency on chardet:

>>> import chardet
>>> detected = chardet.detect(r.data)
>>> detected
{'encoding': 'ascii', 'confidence': 1.0, 'language': ''}
>>> r.data.decode(detected['encoding'])
'User-agent: *\nDisallow: /deny\n'

So urllib3 leaves the user to decide how to consume the data it returns, whereas requests provides a little extra sugar for common use cases, but in so doing introduces some implicit behaviour.

Conclusion: for most content, the developer experience is pretty much the same; for simple text use cases, requests provides a little extra syntactic sugar.

Posting some content

Next, we’ll post some content to a server. And again, we’ll look at three variations of this: query parameters, form data, and JSON content. In all cases, the responses will come back as JSON, and we’ll show only the relevant parts of those responses.

So, starting with query parameters, our examples look like this:

>>> uri = "https://httpbin.org/anything"

>>> r = get(uri, params={"name": "Alice"})
>>> r.json()
{'args': {'name': 'Alice'}, ...}

>>> r = request("GET", uri, fields={"name": "Alice"})
>>> r.json()
{'args': {'name': 'Alice'}, ...}

Once again, there’s basically no difference here, except for a bit of naming. So let’s move on to form data:

>>> from requests import post
>>> r = post(uri, data={"name": "Alice"})
>>> r.json()
{'form': {'name': 'Alice'}, ...}

>>> r = request("POST", uri, fields={"name": "Alice"})
>>> r.json()
{'form': {'name': 'Alice'}, ...}

In this example, urllib3 actually contains slightly fewer lines than requests. But only because we make an extra import in the requests case (the post function). With urllib3, we continue to use the request function that we already have. And we could do the same with requests if we adopted their “advanced” approach.

One subtlety which you may not have noticed is that requests requires content to be passed via the params argument for query parameters and the data argument when working with form data. Urllib3, on the other hand, uses fields for both, which could help to reduce mental overhead for users. But as with all these things, it’s a small difference.

Finally, we’ll look at posting JSON content:

>>> r = post(uri, json={"name": "Alice"})
>>> r.json()
{'data': {'name': 'Alice'}, ...}

>>> r = request("POST", uri, json={"name": "Alice"})
>>> r.json()
{'data': {'name': 'Alice'}, ...}

In a similar vein to the response .json() method, the json argument was added to urllib3's request function in version 2. And so ultimately, both libraries end up implementing JSON support in the same way.

Conclusion: with a few very minor naming differences aside, both libraries offer almost identical approaches to posting content to a server, regardless of the content type.

And the final conclusion

We haven’t explored every single detail of both libraries. Far from it. That would require a much longer post, and a lot more patience on my part. But we have looked at some very common use cases.

Requests sells itself as “an elegant and simple HTTP library for Python, built for human beings”. Elegance is somewhat subjective, but as things stand today, there aren’t many (if any) examples where requests is more elegant than urllib3. Perhaps the one case where this could be argued is when handling textual response content. But this “elegance” comes at the expense of explicitness, which is a guiding tenet of Python.

In terms of simplicity, requests might have a slight edge for less technical users. But there isn’t much to write home about there either. It’s likely that there aren’t many human beings who couldn’t work with urllib3 just as easily.

So which HTTP library should you pick for your new Python projects in 2022/23? From the examples here, it’s hard to conclude that either is objectively “better”. Both actually offer an incredibly similar experience. They require similar amounts of code, and expect a similar level of pre-existing HTTP knowledge. Although requests certainly offers a couple of extra spoonfuls of syntactic sugar.

Perhaps there are deeper use cases that we haven’t looked at here, for which requests really improves the developer experience. Or perhaps recent work on urllib3 has closed the gap between the two, and requests no longer adds any real value.

Does anyone really still need a wrapper around urllib3?