QUIC and HTTP/3

Nick Shadrin, nick@nginx.com

• Professional in the area of Internet systems, the web, HTTP protocols;
• Product Manager at NGINX, the company behind the most popular web server technology among high-performance web sites / now busy with software architecture, mostly control-plane solutions, orchestrating large with a lot of different web servers and systems of all kinds;
• One of the pioneers of NGINX Unit — a fully dynamic web and application server launched in 2017.

Introduction

An upgrade to HTTP/3 is going to be a significantly more complicated and significantly lengthier process than that for HTTP/2 or any other versions for several reasons:

• functionality of QUIC and HTTP/3,
• difference in transport protocol (UDP),
• encryption,
• how TLS is working there, and
• big difference between how it’s working in UDP and in a protocol.
• connection ID functionality and the distributed nature of working through these datagrams in one connection.
• real-world implementation.

Basics of HTTP

As very few people use HTTP/3 today in their environments, it is useful to backtrack to the basics of HTTP/1 and HTTP/2 before evolving into how HTTP/3 got formed and created. The basics stay the same in all HTTP protocols:

• request,
• response,
• HTTP methods,
• URLs,
• header definitions,
• header values,
• body of the request, and
• body of the response (if any).

HTTP/3 keeps the same semantics, but it is placing them in a different transport. And it’s not as simple as it sounds.

Main differences between HTTPs

For the last 30 years, we have been working with different transport across the protocols.

HTTP/0.9

In the very beginning, there was the version HTTP/0.9. If you open Telnet to port 80 of a website that does support that protocol, for instance, nginx.org port 80, type GET/ and press Enter, you will receive the content with no headers, no ways to manage that connectivity and to tell the server to do something else other than just give you the default page. In some cases, you will receive a response from a CDN or somewhere else that will say “That version is not supported” or just “Please don’t use HTTP/0.9.”

HTTP/1

HTTP/1 was created with the functionality of headers and different other features, such as different options for managing the methods, the contents, upgrades, etc. HTTP/1.1 was the one that added more features for HTTP headers and ways of controlling the connection. HTTP/1 (as well as HTTP/1.1) was only using one TCP connection for one piece (object) of content, for instance, CSS files or Java scripts or other things on a page.

By default, the browser opens five or six connections in parallel trying to get all the different parts of content at the same time using that number of TCP ports meaning different TCP connections. However, in the real world, there were some limitations as opening too many TCP connections slowed down some of the network equipment and web servers.

Around 2012–2013, SPDY was invented — the project where Google tried to put different HTTP requests inside of one TCP connection to make it possible to utilize more of the TCP bandwidth of the network. SPDY evolved into the standardization of SPDY in HTTP/2 protocol.

HTTP/2

HTTP/2 protocol is different from SPDY as it uses different methods of compression for HTTP headers and contains things related to its negotiation. So, HTTP/2 got quite a bit of adoption, but that adoption was quite slow. In addition, HTTP/2 was not performing well in networks with a lot of jitter and packet loss.

It turned out that if you have high packet loss, using multiple HTTP/1 connections was better than one HTTP/2 connection. For the ideal networks in terms of packet loss but high latency, if your ping is 400 milliseconds, but you have good connectivity of all the packets reaching the destination, the HTTP/2 was doing really well, as well as with a lot of smaller objects on the web page, for instance, with just one HTML, one CSS, and one Java script on the page. With, for instance, 100 little pictures and 20 CSS files that can be compressed well, and another 50 APR requests on the page, HTTP/2 was also performing well because, for each of those little requests, there was no need to reestablish the connection to all the negotiations anymore. So, it was possible to use the same TCP connection and different streams inside of that TCP connection in order to make HTTP requests (one HTTP request per each stream, but all of the lower-level network optimization was working on one TCP connection).

From the standpoint of low-level network devices, there wasn’t much difference between handling port 443 with HTTP/2 or HTTP/1. It was still the same kind of port, the same kind of connectivity, the same kind of lower-level optimization in the devices on the way.

HTTP/3

In HTTP/3, UDP transport is used instead of TCP transport. UDP is operating basically the same way across different operating systems. It is a much lower-level method of passing packets through the network, the higher-level functionality is needed further up in the protocol.

NOTE. HTTP/3 handles negotiation like TCP. There are no built-in things that mitigate negotiation like TCP or that do the UDP kernel stack.

QUIC

In QUIC, the big difference is that we need to implement encryption features within the QUIC protocol though most of the content might be cat pictures on the Internet and nothing else. For some reason, in HTTP/2 a decision was made by the browser vendors not to support plain text versions of HTTP/2. And the same vision continued with HTTP/3, with even more attention to encryption. So, encryption got way into the protocol, and there’s no way to get out of that.

With HTTP/2, there was a way to create plain text with HTTP/2 connection. The plain text worked from the concept of Client and Server, but realistic browsers, like Chrome, Firefox, etc., would not support that possibly because of advertisement companies or spoofing of plain text connections.

HTTP stacks

If you compare stacks:

• with HTTP/1, TLS and encryption of all kinds were optional. When cURL is used and a page is got, the data is got directly, and it is readable.
• with HTTP/2, we have streams that are all kinds of binary data, which is now encrypted, but it’s still working on TCP.

To look into HTTP/2, we needed to start using different tooling and things like cURL, web browser development tools, and others. They had to be updated around the timeframe of 2015–2017 in order to support the works of HTTP/2 protocol, and it was a long and painful process to get the update through all the variety of operating systems. Now we have the same issue — updating the tooling in order to read HTTP/3,.

Reality of HTTP deployment

In the realistic environments, the support of HTTP/1, HTTP/2, HTTP/3, and whatever other extensions is used on the very front of the network where you connect to something like NGINX or other servers. So, there wasn’t much demand for the back end.

In the back end, you usually have a very nice high-quality network, so latency is not the issue. In the back end, you’re looking at doing specific requests to specific servers instead of having a variety of picture requests where you don’t care which picture is going to be downloaded and so on. Back end actually turned out not to be very demanding for such challenges. So, the back end is HTTP/1.

NOTE. If you have HTTP/2 or HTTP/3, or both, or whatever in the front end of your network, please don’t ever turn off HTTP/1, or HTTP/1.1. Google search engine pessimizes web pages if they don’t have HTTP/2 support. It elevates the pages with HTTP/2 support, and the website goes down if you are not using HTTP/2 and only using HTTP/1. The punchline: in order to scrape those sites, Google uses HTTP/1. So, for instance, if you want to get rid of the DDoS attacks, turn off HTTP/1, but you will also lose the search traffic or some other clients that still use HTTP/1.

HTTP/3 benefits and issues

UDP as a transport

Since we are using UDP transport and implementing features of encryption, congestion control, etc. on the application level, we have less reliance on the kernel. It matters if you manage both the clients and servers by yourself. In some cases, while using HTTP/1 and HTTP/2, you are unable to create the piece of a client’s software because it will be looking at the TCP stack, while the client is still using the old operating system there. It will use the kernel features of a TCP stack. They are the same regardless of what you want to do on the application level. However, in HTTP/3, if you are creating the server and the client, in the client you can do all things that you like and still use Windows XP or whatever.

UDP is supported in the same way. This can actually be a bad thing as since you are relying less on the kernel, you have to build more things yourself. It’s more work, and the question is if you are going to do it better than people who already did it within 30–40 years of TCP development?

Encryption differences

The built-in encryption in the protocol allows for taking it out and having a different encryption style from your normal OpenSSL, OpenTLS, or other things around your machine.

This may be a benefit. If you are building your own BoringSSL or LibreSSL, or QuickTLS, or whatever other library, you have to look at that one in addition to your OpenSSL. Still, we are expecting more SSL or encryption vulnerabilities in the world. And if you’re using HTTP/3, you have one more encryption library to worry about, which might or might not be better than your normal OpenSSL however.

Connection ID

The next very important functionality is the connection ID. In layman’s terms, we don’t have to look at the IP addresses to combine a bunch of datagrams going into your server into logical connections if we have the connection ID.

If I’m trying to access a web page over the phone, some of my traffic might go through my 5G connections, some — through Wi-Fi, or the IP addresses are changing all the time. In theory, this should work with HTTP/3 without breaking the logical connection. Anyway, changing connections and still being able to maintain logical connectivity to the server is a very promising idea.

NOTE. On UDP, there is no handshake mechanism. The main idea is that there is a connection ID coming inside of the protocol. But in the ideal world, your traffic can be coming from different IPs, but still be the same kind of logical content, so the IP verification done in HTTP/3 is not necessary.

Faster negotiation of the HTTP protocol

There are different ways of negotiating HTTP protocols.

• negotiation from non-encrypted HTTP to encrypted HTTP (the simplest method), for instance, within the protocol or even within the web page itself using the HTML tags or JavaScript functions as well, and be it 301 redirect or 302.
• negotiation from unencrypted into HTTPS using the strict transport security headers telling the browser to not look at unencrypted websites at all and to remember for a long time that the website only works by HTTPS and shouldn’t use an unencrypted connection to that site. This is not a strict convention and some browsers might lose that header or clear it for whatever reason, so the “strict” transport security is not that strict.
• HTTP Upgrade — a special HTTP convention where the most usable thing is how the standard HTTP connection is upgraded into a web socket. In order to upgrade into a different protocol, you first need to establish a connection with the previous protocol.

Methods of negotiating a protocol between HTTP/1 and HTTP/2

The first, easy, and logical way of negotiating HTTP/1 into HTTP/2 is to use the upgrade header, that is basically to tell the server that you support that protocol, and the server will be giving you the one hundred continuants, and so forth. But the thing is you are talking too much over HTTP/1 before you are starting sending content over HTTP/2. For that reason, there were different methods of connection negotiation created between those protocols, for instance, NPN (Next Protocol Negotiation), and also ALPN, both of which are extensions to TLS.

HTTP/2 has TLS connectivity, can be encrypted and unencrypted. If HTTP/2 is properly encrypted, the only method of negotiation of HTTP/2 that was adopted by browsers, such as Chrome or Firefox is ALPN. ALPN is very simple: you are sending your initial packet to the server, and with that packet, you are saying that you accept HTTP/1 and HTTP/2 connections, and the server starts responding immediately with the keys. If you had preshared sessions, you immediately get the content coming back straight up without extra hops to negotiate protocols. So basically, you are sending your protocol knowledge immediately as you are connecting to the server. In this case, HTTP/2 inside those ALPNs is called h2. If you are doing some Wireshark troubleshooting, you will see those h1 and h2 just inside your TLS extensions.

Alt-Svc header

These methods don’t work with HTTP/3 as we have not only to negotiate the ways of how we are sending that traffic but also to change the method of transportation of that traffic, which now is UDP instead of TCP. You have to go to some different type of connection establishment. For that idea, the thing that got standardized is the Alt-Svc header.

1. The first method is working for HTTP/2 just as much as HTTP/3, and so it got standardized not only for HTTP/3, but for other protocols as well. In this case, we are defining the hostname and the port, and the max-age parameter — the ma stands for max-age — the time in seconds, for which your browser or your client is supposed to remember that you have that type of connectivity available in order to perform faster negotiation, in the subsequent further request later and later.
2. You can also do it for h3 with a different port.
3. You can define it without the hostname, which means connecting to the same thing using a different port. It does not have to be a different port. In most cases you will be using HTTP/3 on the same port 443 like you are using with a normal HTTP/1 or HTTP/2 connection. But for any other things or reasons, you might want to use it on a different port, whatever suits your specific environment. But UDP ports might be blocked.

HTTP/3 version negotiation

So, HTTP/3 provides faster negotiation, but only in case of an “optimistic negotiation.” Basically, some clients, some browsers might want to send the TCP request and also the UDP request at the same time. The faster negotiation is supposed to be working, but it is not specifically the convention. You might call it the prior knowledge protocol negotiation.

HTTP/3 optimistic negotiation

Real-world environments / Challenges

Migration to HTTP/3

Infrastructure challenges

In theory, when the client is talking to your server, you have both of those ends very well known, so everything is great. But in the real world, in between the client and the server there are things that are not necessarily the ones that you know and own. It might be your application firewall or a load balancer even down to the switches and routers of the providers. In reality, you don’t have the client connecting to your server, but the client connecting to a load of boxes, and only then one of those boxes will be connecting to your server. And all of that support of different HTTP protocols needs to be either properly implemented in between or totally ignored in between. Which is better?

This is the choice between usability and security. If you want everything to be absolutely secure, you need to decrypt everything at every step. All of those spenders will want to dig into every bit of your traffic and every logical piece of your traffic. That doesn’t really work in terms of performance and in some cases in terms of usability. But the total ignorance of the traffic also doesn’t work. Some of those boxes right now know that traffic well, some — don’t. That’s why the migration process into HTTP/3 is going to take a little bit longer than it took for HTTP/2.

Server engineer challenges

Another challenge for NGINX and other companies developing HTTP/3 support for software or hardware is that the UDP stack is not as optimized as TCP. This sounds counter-intuitive because of TCP, but much effort was put over the last decades into optimizing TCP. While no one has really done optimization to UDP, once we have started implementing features of UDP performance into the protocol and back into the application. With TCP you have legacy, while without the legacy with HTTP/3 and UDP you can implement new features in there: use newer kernels, newer lower-level hardware functionality, etc.

If we go deeper into the web server mechanics, there are multiple processes there, and normally UDP packets are going all over these processes. We need to recombine them, use some kernel features sometimes, However, making UDP transport for a web application was not an easy task.

Tooling challenges

The lack of plaintext version and the minimum debugging features are another problem. For instance, you start working with some HTTP/3 website with the Chrome browser, and something breaks somewhere in the middle. The browser will go back to HTTP/2, HTTP/1, back into TCP and decide “OK, that UDP doesn’t work. I’m going to forget that this website works with HTTP/3, and I will go down to HTTP/2 and HTTP/1, and not use HTTP/3 at all.”

To get your browser back on track to use HTTP/3 again, there is, for instance, an option for the kernel to force the use of HTTP/3 on a specific domain name. However, this is more challenging for all of your customers. If you lose a little bit of your HTTP/3 connectivity, all of your network basically goes to HTTP/2, and only some of the customers will come back and there will be a mix of protocols. In reality, HTTP/3 will not work 100% of the time with 100% of your users. Hopefully, it will work with a large percentage of users, but the number of them will be specific for your project.

Security challenges

In relation to security DDoS attacks, UDP is not very trusted. Still, if you turn off UDP at whichever level, you’re losing all of your HTTP/3 connectivity entirely. It is a solution to get rid of a DDoS attack. However, if your network was so much optimized that you cannot actually handle TCP there, you just turned off UDP and you got the DDoS attack back by your old users. Turning UDP off is not a silver bullet in performance as you still have to be able to maintain that TCP connectivity back.

Agility of the protocol is another important issue. A lot of changes with a piece of software, especially with protocols, make it hard for all of those devices in between your server and your client to support all of those changes and all of the agility of the protocol. So basically, you need to wait for the protocol to be more stable in order to be implementing that in the environment.

Implementations in NGINX

NGINX has a separate branch supporting this protocol. At quic.nginx.org you will find a How-To on how to install it, how to get that branch, how to compile it, and what to do next. Soon, we’re going to have it in the mainline. However, OpenSSL APIs and OpenSSL functionality doesn’t allow us to build QUIC on HTTP/3 on top of OpenSSL easily, and that slows us down. Now, the only way to support HTTP/3 at NGINX is using it with BoringSSL.

BoringSSL doesn’t exist in pretty much all of the mainstream operating systems as a standard package. You have to find it yourself, compile it yourself, and make it yourself, that is first compile and install the library, and then you install that special ‘NGINX with HTTP/3’ module that will be linked to that library.

It is technically possible to compile a static version of NGINX with all the encryption features, but it will change the product, and it will change the life cycle, and your response to the security vulnerabilities. We believe that encryption should be separate from the web server software, so we are against including encryption features inside of NGINX. This is one of the reasons why it is hard to do the mainline release with HTTP/3 features, as security and encryption functionality and BoringSSL are not a part of normal packages.

• It is pretty simple to configure NGINX for HTTP/1 — ‘listen 443 ssl’.

• For HTTP/2, pretty much all of your NGINX versions are configured safely, if you don’t have the HTTP/2 parameter in your ‘listen’ directive.

• With HTTP/3, we need to add the Alt-Svc header and to create another listener for that HTTP/3 protocol in addition to the normal TCP listener.

NOTE. This doesn’t work with stock NGINX as of December 2022. It will be working with a special branch upcoming in the mainline.

NOTE. NGINX can be forked by anyone, which is important in the world of open source, especially for the companies limited in their ways to purchase software. One good example of a successful NGINX fork is OpenResty done by Yichun Zhang. Another good example was Tengine done by Taobao in China. In addition, a Moscow team did a very recent fork called Angie. With some legal restrictions, you might be using this software instead of America N5 software, or America N5 NGINX software.

As for fragmentation of the NGINX environment, especially in the context of OpenResty and Angie, the developers from Angie mentioned that they want to be as close to NGINX mainline as possible for as long as possible, which is the correct way of collaborating in the software like that. It’s better for us to be maintaining a closer code base between those projects and forks and be able to exchange the greatness of the new code for the benefit of the whole community.

Conclusion

• One of the main premises here is that HTTP/1 is not going away, as it’s still well-suited for backends and application runtime.
• HTTP/2 is basically the standard for Internet-facing web-services and protocols, though it failed to deliver on promises and still needs fixing.
• HTTP/3 with QUIC addresses many HTTP/2 challenges, but HTTP/3 is still under testing now and is expected to be Internet-facing-only for some time.