How HTTP proxies read TLS encrypted traffic from browsers ?
In section 1 of this article, I have provided an overview of why HTTP proxies such as Burp Suite or OWASP ZAP are able to show us TLS encrypted HTTP protocol level data. The second section (aimed at security engineers or consultants) explains why the proxy interception works flawlessly despite modern security mechanisms such as HTTP Strict Transport Security (HSTS) and HTTP Public Key Pinning (HPKP).
Section 1
One misconception people might have about interception proxies is that they somehow “break” the TLS protocol to observe the traffic. Technically, that is not true and we will shortly see why. Let me start the explanation with a diagram given below :
Henceforth, I will refer “TLS encrypted HTTP data” as “HTTPS data”. There are actually two TLS connections established whenever we are able to see HTTPS data in an intercepting proxy. Before the actual TLS handshake occurs, the server’s identity (public key) is verified by clients (browser or mobile devices). During the initial stages of a TLS handshake, the server sends the certificate to the client which contains its public key , the domain for which the certificate is valid and many other details. This public key of server plays a very important role in future handshake messages. However, in this article, I am not going to dive deep into cryptographic details of a TLS handshake. I might write another post explaining a TLS handshake in detail using packet level data.
So, there has to be a way for the client (browser or mobile devices) to know that the certificate was indeed sent by the intended server and not some adversary trying to perform a Man-in-the-middle (MITM) attack. The TLS protocol itself does not specify anything related to verification of server’s identity (public key). The protocol is also not vulnerable to MITM attacks. Clients rely on something called as Public Key Infrastructure (PKI) to verify the authenticity of server’s public key. To explain PKI in short, let me first show you another screenshot of reddit.com certificate :
Simply put, the certificate of www.reddit.com is cryptographically verified by “DigiCert SHA2 Secure Server CA” . Certificate of “DigiCert SHA2 Secure Server CA” is in turn verified by “Digicert”, which is called as a root Certificate Authority (CA). All clients (browser and mobile devices) have some root CA certificates by default. Clients trust these root CAs implicitly. This is where we fool the browser when we perform TLS interception. As you might know, we install a certificate provided by the interception proxy in the browser (or mobile device) to seamlessly see all data sent and received by a client. Now, let us see the certificate of www.reddit.com when an intercepting proxy is enabled :
Observe that the browser did not show any warning about the TLS connection being intercepted. Also see the root certificate authority name. Here, we fooled the browser by installing a fake root CA certificate in its certificate store. By fake , I mean the one which is not shipped with it when you download it. Thus, with the intercepting proxy enabled , each time you visit any HTTPS website, the proxy provides you with a fake certificate for that website signed by the “fake” root CA certificate. Browser believes that it is indeed talking to actual reddit server and happily proceeds with HTTP communication. This is the story of TLS connection 1 in Figure 1.
The second TLS connection (TLS connection 2) is a normal TLS connection. The proxy acts as a client and connects to the server using TLS. After this, the proxy keeps forwarding data back forth between these two encrypted channels so that you can play with it. If you want to deep dive into TLS protocol and PKI , I highly recommend this book.
Section 2
All security engineers with knowledge about TLS and PKI know the purpose of HSTS and HPKP . I will still state it in brief.
HSTS: HTTP response header used to tell browsers to always connect to a server using HTTPS only (in future) and also prevent connection if the browser encounters any error during certificate verification process.
HPKP : HTTP response header used to tell browser which public keys should be present in the certificate chain while verifying the server’s identity. This is done to narrow the trust to only few certificate authorities.
So, the question is , despite HSTS and HPKP protection on website such as Google and facebook (even when preloading feature is present in some browsers), why the intercepting proxy works just fine? I found the answer in this article. It is because browser ignore HSTS and HPKP errors when the root certificate has been locally imported. Those protections are for an over-the-wire MITM attacker. If malware installs a root CA certificate in your browser, HPKP and HSTS won’t protect you.
I always welcome constructive criticism of my articles. Please let me know if you have any questions or suggestions. My twitter handle is @ EthicalEvil. Thank you.