Revived the promise made six years ago for Requests 3

Requests is undeniably unmissable for anyone willing to start in the Python ecosystem. Yet the community is broadly unaware of how sad the situation has become.

The incident

Due to dramatics circumstances, it was partially locked and put in “maintenance mode” only, meaning that it would no longer evolve (features speaking).

Some of you may have eared, more than six years ago that Requests was working toward a major version, namely the V3. There was even a completed fundraiser with a substantial amount, around $30k.

At the time, there was a plan for bringing Async, and HTTP/2 support (most notably). You’ll guess that none of exposed promises ever landed.

Impacts

This incident led to several impacts.

• Maintainers felt too much pressure on making steps toward a major, some of them said: “We just can’t deliver the promises, and it’s going hard to explain why”.
• All the back pressure landed at urllib3.
• Abruptly started to refuse any changes, saying: “Requests is feature complete, and it’s popularity makes it nearly impossible to fix bugs that may be considered feature to this point”.
• Newer project started ditching HTTP client Requests by learning to use a new one.
• Some large scale projects bound to Requests did not migrate because of the potential high cost doing it.

Let’s be clear, this article is not about saying “Requests maintainers are bad” but rather saying we don’t agree, and everyone is entitled to his choices.

For many years now, Requests has been frozen. Being left in a vegetative state and not evolving, this blocked millions of developers from using more advanced features.

The Journey into Waking Up

Initially, like many other reasonable person, I said to myself “Let’s wait, Requests will eventually unblock the situation!” Yet the years are still passing and nothing.

Let’s realize something, Internet Explorer 11 just celebrated its tenth anniversary last year, and you’ll never guess that it has support for HTTP/2 integrated! Yet Requests is unable to do so.

I said to myself: “Why are we waiting for this indefinitely? They don’t owe us anything! There’s a reasonable chance we are waiting… traped in a circular reference.. Let’s act.”

Then, I tried to get a firm grip on urllib3 base code, contributing this and there until I was ready to kick things up with a proof of concept that would have put urllib3 far ahead. Without any breaking changes. I was delusional. This was a bit of a shock, but six months passed between my initial kick off and my formal give up, and here’s why in a nutshell:

• Every bits of idea brought to the table was dismissed, and never seriously analyzed.

This felt as good as taking the coldest wind on your face while exiting your warm home. It’s the hard truth of the OSS, it’s not a democracy and it’s not ruled by pure logic, but humans.

No far latter on, urllib3 called out a fundraiser to help with HTTP/2 support (only) and claimed around $40k to be able to deliver. They probably meant well, but[…]

This was the last straw for me, knowing the past mistakes, and actually the best incentive to break the cycle. Everything point to another year before having the slightest sign of a PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n into our production pipes. They clearly lack the time to move forward (or incentives? or both?). No worries, as developers, we should share this burden for the greater good.

From my perspective (my opinion only), the OSS ecosystem should not fund raise money for a decade old protocol, and most of time should deliver prior to asking. Entities raising money with no solid proof of concept, that’s rare!

This felt like a bit of a hostage situation, but it could only be me seeing it this way.

The Solution

So many years and expertise have been brought to Requests and urllib3.. Do we throw all of it into the garbage disposal? Engineers should be reasonable scientists and not zeled revisionists.

We don’t have to reinvent the wheel all over again, HTTP client Requests is well established and really pleasant in its usage. We believe that Requests has the most inclusive and developer friendly interfaces.

So how to propose an attractive continuity for Requests? How hard could this be to (i) bring a true upgrade experience and (ii) keeping the compatibility as high as possible?

Sometime the OSS ecosystem act like it is trying to land a rover on the Moon, let’s be more realistic and bring our vision down to firm grounds.

There’s a chance offered to us to rehabilitate Requests (and urllib3) and give a fresh breath of air to us, the developers. Without having to learn something as basic as handling a HTTP client.

So, the work started, without looking back.

Just around my intent to give up on urllib3 evolution, we proposed a fork of both urllib3 and Requests, namely urllib3-future and Niquests.

They are (truly) drop-in replacements, at the time of writing, those forks are six-months old! And the whole work around this is roughly a year old. We listened to what the community needed by… triaging all the issues on major repositories (Requests, urllib3, and dependents of them) so that the upgrade would be truly useful.

Are you seeing it coming? Great! Let’s look at the feature table comparison against requests, httpx and aiohttp!

And you would think… performance..? are bad?

Look at the performance comparison against them!

Scenario: Fetch a thousand requests using 10 tasks or threads, each with a hundred requests using a single pool of 10 connection.

High-Level APIs

Simplified APIs

Did you give up on HTTP/2 due to performance concerns? Think again! Multiplexing and response laziness open up a wide range of possibilities!

And everything you are witnessing is in fact “Requests”-extended and you have nothing to learn again to get started. We kept 98% of the test cases from both Requests and urllib3, then extended them to cover newer features.

We are looking forward to see you at the Niquests repository! Let’s say it’s a promotional offer, get everything and more for $̶4̶0̶k̶ $0!

• #1 You depends on Requests? Discover Niquests instead.

Drop-in replacement, simple patch needed, tiny effort required.

• #2 You depends on urllib3? Discover urllib3.future instead.

Drop-in replacement, no patch required, atom-like sized effort required.

By visiting the GitHub repository and documentation you will be able to learn more about this, and hopefully make a decisive switch!