ESNI & DNS over HTTPS will change networking — here is why.
Two weeks ago, Apple, Cloudflare, Firefox, and Fastly, quietly implemented and published ESNI (Encrypted Server Name Indication). Discussions at public mailing lists indicate that very few people noted, and less realize, the potential ESNI (along with other similar efforts like DoH) has to utterly shift the way network policies are defined and enforced into the network. This posts discusses some of them.
So what is SNI? The URL of the destination server you are heading to (e.g., https://medium.com), sent unencrypted in cleartext, even when you use HTTPS. It is necessary, so that a server that hosts multiple domains can send back the certificate for the desired domain. As you can easily guess, ESNI is an encrypted version of the SNI. The server can still decrypt it and take the right action, but nobody else in the between (i.e., no network) has visibility into it.
Cloudflare’s announcement goes deeper into the technical details, but the main gist is that ESNI prevents in-path third parties (aka networks) from seeing which websites/apps people visit. As Matthew Prince, Cloudflare’s CEO, puts it, ESNI fixes one of the core internet bugs.
Privacy VS Network Policy: Past, Present, Future
To better explain the implications of ESNI on network policies, I’ll look at two conflicting trends from a past-present-future perspective:
- Privacy: how much of a users’ personal information is exposed when online?
- Network Policy: what content-aware network policies can operators define and enforce?
The Past (aka 2006)
It was pretty hard to hide what you do back then. Most traffic was unencrypted. Any network that forwarded your traffic could monitor websites you visit, videos you watch, articles you read, just by looking at your actual traffic. Typical in-network policies were firewalls, content optimization, QoS, zero-rating, and DPI-based user profiles. A wide group of vendors made sure that necessary tools were baked into network appliances, and network operators could define and enforce their policies without any coordination from endpoints.
The Present (aka 2018)
We are at a transitional phase. The actual content of user activity is largely protected due to adoption of HTTPS. But a whole lot of information is still leaking through DNS and SNI. Network operators can still derive which websites and apps you visit and how frequently. The content-aware network policies we see today are around corporate firewalls, censorship, decryption/inspection, zero-rating, bandwidth management, and user profiling. They are typically enabled by a DPI-like appliance in the network. There are still ways to bypass or hack these policies through VPN/Tor, but the burden is on users, and a very small minority pays the extra monetary, performance and setup cost to do it.
Now, there is an interesting trend happening: while the network can largely derive what you do, doing this real-time, accurately, and with no disruption is challenging. All cloud-based services use CDNs and datacenters to deliver their traffic; they frequently change their IP addresses and URIs, and this interferes with network policies in the network, which leads to problems. To give you an example: Facebook has partnerships with 100+ mobile carriers worldwide to zero-rate its traffic (i.e., users don’t spend their mobile data when they visit Facebook). If Facebook were to add a new URI without telling their partners, users with no data would not be able to visit Facebook, and users with data will see their mobile data consumed. It would take days or weeks for a network operator to detect the problem and reconfigure the network, users would complain to both Facebook and their carrier, and nobody would be happy.
To avoid the hassle, large content providers carefully structure their backend and coordinate with network operators to ensure jointly beneficial network policies work. For example Facebook maintains large backend proxies for its zero-rating partnerships; Enterprise software services publish their IP addresses and URIs to help network operators configure their firewalls (examples from Microsoft and Salesforce); and more than 800 content providers participate in 150+ zero-rating carrier services, where humans manually exchange IP addresses and URIs to keep these services up and running.
Summarizing today’s status: our online activity today is exposed by default; network operators can enforce a number of policies using SNI and DNS, but manual coordination with content providers is needed to achieve good accuracy and policy uptime.
The Future (and how we’ll get there)
So what does the future hold, and why is ESNI so important? ESNI, combined with DNS over HTTPS will make our online activity opaque by default. This is a huge win for privacy, undisputedly, and rightly all news attention went to it. But it will also have big implications on what policies network operators define and how.
Networks traditionally define content-aware policies by implicitly inferring context from the traffic itself (SNI and IP). As privacy and network controls march in reverse directions, content providers and networks found a sweet spot between privacy and network policy: networks know that a user streams Spotify every day, but they don’t know which songs they do. This changes with ESNI, as it pushes the privacy trend one step forward.
Fast forward to the future, and by default, networks will know nothing about what we do online.
Will this happen overnight? No. Will content-aware network policies extinct and networks will just forward packets? No, by no means. Firewalls, zero-rating and other services generate significant business and user value for people to let go.
What we will observe instead, is a transition from a world where everyone can implicitly infer what you do, to one where context is securely exchanged between mutually trusted parties.
At the beginning we will see two types of content providers:
Providers that prioritize coordination with networks: A company like Salesforce will likely decide to stay detected. They will want to ensure that no corporate firewall blocks their traffic, and at the same time feel that nobody bothers if networks know they use Salesforce. These providers will keep doing what they do today — structure their services in an easy-to-detect way.
Providers that prioritize user privacy: PornHub will want to go opaque to enable their users watch porn without worrying that somebody spies on them. These providers will jump to the privacy train (using ESNI, DNS over HTTPS, and shared IP addresses) as quickly as possible.
As adoption increases, a third approach will evolve: providers that stay opaque to protect their users privacy, but coordinate with specific networks for jointly beneficial services. For example, Twitter will want to stay opaque in countries where censorship is likely to happen, but be detected by a mobile carrier that zero-rates its traffic.
This third approach will eventually become the dominant one and replace today’s status quo. For this to happen we will need wide support for related protocols (ESNI & DoH) in browsers and mobile network stacks, along with a scalable way to explicitly exchange context between endpoints (clients, servers) and trusted networks in a secure way that respects user privacy. Having a proper way to talk to the network, will also lead to more interesting and accurate network policies. For example an enterprise service from the first category will share finer granularity context with a network (e.g., this is VoIP call from the sales department that needs to be prioritized).
This transition will be driven by companies that facilitate a significant volume of user activity. The companies behind the individuals that drive the standards are a representative set (Apple, Cloudflare, Google, Firefox, Akamai, etc). Networking companies will come later, and DPI vendors — like Sandvine and Allot — that differentiate themselves by the number of apps they can detect will be challenged.
Exciting times ahead! Interested in these ideas? We are thinking day and night about these at Selfie Networks — drop me a line and I’d be happy to chat!
