Know your CDN & Protect your Origin

Published in

Axel Springer Tech

12 min readJul 15, 2020

For running larger websites, CDNs can be advantageous to improve delivery efficiency by reducing latencies and increasing bandwidth to clients using distributed caches.

A basic CDN setup: Users retrieve a website’s content through CDN servers, which fetch it from an Origin

As CDNs do not generate content (but rather only deliver it), its servers connect to a usually comparably small number of servers which generate content or store the original versions of static files with javascript code, cascading style sheets, images or other media.

These servers are commonly called the origin servers, as they represent, from the perspective of the CDN, the source where content is fetched from.

This article discusses some additional challenges arising from this setup as well as two main approaches to solving them.

Identifying the CDN and the Actual Client

To make best use of a CDN, one might want to delegate some authority to the CDN, like checking authorization to access paid content. Also, one might need to know information about the client which originally connected to the CDN.

Access through the CDN and directly to the Origin: How can we identify which is which?

Adding some headers is usually trivial and CDNs can be expected to enrich requests by default, for example by adding the connecting client’s IP address to the non-standard but widely used X-Forwarded-For request header.

Yet we can not trust these headers by themselves, any HTTP client could have added them and created a request to our origin. So in addition to enriching requests with additional information, we need some trust anchor, some proof to know if requests which look like they came from the CDN actually did.

The Potentially Vulnerable Origin

While I would recommend to have an origin with an own powerful, efficient http cache infrastructure in order to efficiently deliver content with or without a CDN, and to provide a high level of resilience against DoS and other attacks, website operators might decide to keep their origin setups minimal and rely on the CDN to provide additional protection.

And there is some logic to it: If you already pay a CDN for caching and efficient delivery, does it not make sense to scale down your origin?

Yet, if you do, your origin becomes even more vulnerable than it already was before scaling it down: If adversaries find your origin, attacking it directly rather than through the CDN might be more effective in order to bring down the site. Thus, we should think about how to protect the origin.

Some Attack Vector Classes

In order to qualify the effectiveness of approaches to Origin protection, I would like to introduce a couple of common attack vector classes.

Layer 3 (IP): Volumetric DoS
This includes the common dumb DoS scenarios where usually a botnet or a horde of hacked servers is used to attack some victim by sending a huge number of useless packets in order to saturate or overload network infrastructure, effectively taking down networks or at least severely impairing them.
This method is commonly combined with some means of traffic amplification, classically exploiting badly configured oder implemented DNS or NTP servers.
Layer4 (TCP): DoS
This class resembles the previous class, but acts one layer up and targets the network stack of servers’ operating systems.
A classic example is the SYN-Flood where operating system resources are exhausted by sending TCP SYN packets from random source addresses and ports.
As operating systems have been improved to be highly resilient against these kinds of attacks, this class has become much less relevant since a SYN flood attack was originally launched against the New York based ISP Panix in 1996.
File descriptor & TLS Resource Exhaustion DoS
This class targets processes handling HTTP requests. For example, a trivial approach might be to open many connections (and possibly sending data real slowly (slowloris attack) in order to use up file descriptors or make the server spend considerably more time on some useless calculation than a client needs to launch the attack.
File descriptor exhaustion can be easily avoided with rate limiting the number of connections per source IP or tying the number of allowed connections to some proof of legitimacy. At any rate, using even millions of file descriptors is not much of an issue any more with the amount of RAM available today.
But TLS-based attacks can be a real issue.
Layer 7 (HTTP & Application)
At the HTTP layer and above, adversaries may also attack by sheer volume or by exploiting bugs or resource intensive caluclations.

The CDN IP Whitelist Approach

A rather simple idea to protect your origin is to only allow, on the IP level, access to it from the CDN, but not directly from other clients. If we know the CDNs IP adresses, we can use stateful IP firewalls or even just simple stateless IP packet filters to drop any traffic but from the CDN.

CDN IP whitelisting: Only CDN IP Addresses can connect to the origin

Obviously, to implement such filtering, we need a list of IP addresses from the CDN operator, like these examples:

Additionally, CDNs might offer to route all traffic through a small(er) number of servers in order to further reduce the number of
connecting addresses. For example, Fastly calls this origin shielding. With Akamai, it is an add-on called Site Shield.

The PROs and CONs of IP whitelist approaches

Our Attack Vector Classes

Regarding our attack vector classes, using a CDN with IP whitelisting reduces some but not all risks for the Origin:

class 1: Origins will still need to protect against class 1 (layer 3) attacks, but can reduce risks by trying to hide the origin (e.g. using IP addresses registered for 3rd parties and not publishing them under well known Domain Names).
class 2: It could be argued that protection against class 2 is not that much of a benefit as operating systems already provide a high degree of resilience.
Also, if the attacker is smart enough to spoof the IP addresses of the CDN, all potential protection against class 2 attacks is lost.
class 3: Offloading in particular TLS attack vectors to the CDN remains a clear advantage, but origins can still be protected by other means, e.g. rate limiting.
class 4: The respective attack vectors will clearly remain an issue also with CDN IP whitelisting. It can be considered trivial to trigger CDN cache misses and exploits will exist with or without a CDN. Filtering HTTP traffic (“Web Application Firewall”) can reduce such risks, but is orthogonal to the CDN IP whitelisting question and is, at any rate, best located at the origin for efficiency (there exist offerings for WAF at the CDN, which then require CDN IP whitelisting to be effective at all).

Direct Origin Requests

Routing uncacheable requests through a CDN does not, in the general case, make any sense from a performance perspective: for content that is not cacheable, a CDN will (except for some rare, exceptional cases with clients in badly connected internet-regions) always add latency, so website operators might wish to still provide some direct access path from clients to the origin.

Yet, with the IP whitelisting approach, this contradicts the goal: If, for example, one server IP was whitelisted only for the CDN while another one using the same network infrastructure was directly reachable, the latter would represent the natural target for an adversary, so all potential advantages of CDN IP whitelisting were lost and the whole idea rendered useless.

So, in general, CDN IP whitelisting only makes sense if all origin networks are filtered and no direct access is possible (except maybe for some well known, trusted source IPs).

That, in turn, requires routing all requests through the CDN, whether or not they can be cached. This brings with it some disadvantages like higher complexity, higher cost and higher latencies.

The Alternative: Signed Requests

So, how can we authenticate a CDN while still allowing direct origin requests?

The answer is signed requests.

By having the CDN sign requests, we can identify it and, for example, whitelist it for paywall content while still allowing it for direct requests to the origin.

By adding a cryptographic signature to all requests going through the CDN, we can reliably distinguish requests made by the CDN from direct client requests.

Signed requests typically contain:

The IP address the CDN is connecting from
The IP address of the client connecting to the CDN
A time stamp
A signature of this data and additional parts of the request, like the URL

The signature itself can, in principle, be generated using any cryptographic hash function, but HMACs are recommended to protect against potential prefix attacks and collisions of the hash algorithm. Used with an HMAC, even MD5 can be considered safe.

Symmetric signatures like HMAC-MD5 or HMAC-SHA256 are significantly (some orders of magnitude) more efficient than asymmetric signatures, for example using RSA. Because, for the purpose at hand, there usually are only two parties involved (the CDN and the Origin), asymmetric cryptography provides no relevant benefit, but would incur a significant performance penalty.

Validation

Validation of requests with signed headers consists of:

A check of the timestamp to be recent (as within the last couple of seconds), thus protecting against replays of signed requests
A check of the CDN IP address
A check of the signature.

Implementations

Signed headers can be implemented in arbitrary formats, but there are some de-facto standards, for example:

My opinion on AWS4 is that it is needlessly complicated, in particular it uses a very peculiar signing key derivation scheme which involves five nested HMAC calls.

JWT are very versatile but require JSON parsing, which implies a couple of additional attack vectors.

Akamai G2O is a very simple format, easily parseable, and the one which we are going to have a closer look at.

Akamai Ghost to Origin (G2O) signed requests

G2O uses two HTTP request headers: One for additional Data to be signed and one for the signature itself. Let’s look at a real-world example:

X-Akamai-G2O-Auth-Data: 3, 95.101.91.77, 134.130.186.117, 1588070051, 8544330.1793717418, 3

This header contains:

A format version identifier (3)
The IP address akamai is connecting from (95.101.91.77)
The IP address of the client connecting to Akamai (134.130.186.117)
A unix epoch timestamp
A unique request ID
and an ID identifying the signing key

X-Akamai-G2O-Auth-Sign: 0FZGHeE0oRK3wu+98UlJpw==

The actual signature is then contained in another header, encoded as base64

G2O configuration

In the Akamai CDN, G2O configuration is, unfortunately, only available in advanced metadata, but can be made available by Akamai Professional Service as a custom behavior for property manager. The basic advanced metadata snippet looks like this:

Akamai G2O configuration

<auth:origin.signature>
<data-header>X-Akamai-G2O-Auth-Data</data-header>
<sign-header>X-Akamai-G2O-Auth-Sign</sign-header>
<version>3</version>
<nonce>3</nonce>
<key>...</key>
<status>on</status>
</auth:origin.signature>

Validating G2O

Implementing G2O Validation is not particularly hard, and there are also quite some existing implementations, e.g.

Please note that we are not using any of these implementations, so this list is only provided for your convenience and implies no recommendations.

Validating G2O in Varnish

To validate G2O in our Varnish-Cache origin servers, we use some additional vmods:

re for regular expressions with back references
blobdigest for hash functions and HMAC support
taskvar and constant from the varnish-objvar bundle for variable support

The VCL code presented here should work with Varnish 6.3 and 6.4. Adjustments to earlier 6.0 versions should, if required at all, be rather simple.

initialization

vcl 4.1;
 
import std;
import blob;
 
import re;
import blobdigest;
import constant;
import taskvar;
 
sub vcl_init {
        # replace badcafe with your secret in hex
        new g2o_1 = blobdigest.hmac(MD5, blob.decode(encoded="badcafe"));
        # nonce-id
        new g2o_nonce = constant.string("1");
        # how much clock skew to accept
        new g2o_tolerance = constant.duration(30s);
 
        # 3 = HMAC-MD5
        new re_g2o_auth_data = re.regex("^3, " +
            "([0-9\.]{7,15}|[0-9A-Fa-f:]{3,39}), " +
            "([0-9\.]{7,15}|[0-9A-Fa-f:]{3,39}), " +
            "([0-9]{10,}), " +
            "(-?[0-9\.]{1,}), " +
            "([0-9]{1,})");
 
        # no config below this point
        new g2o_ts = taskvar.time();
        new g2o_delta = taskvar.duration();
 
        new client_verified_akamai = taskvar.bool(false);
}

For our G2O validation VCL, we first of all import all required vmods.

In vcl_init , we initialize

g2o_1 as an MD5-HMAC signer for our secret (the same as configured in Akamai)
a constant variable for our nonce ID
and a constant variable to define how much of a difference between our local time and clocks at akamai edge servers we are willing to accept before we refuse signed requests as “too old”

You will definitely need to change the HMAC secret and probably the nonce ID.

The remainder of vcl_init should not need adjustments: We initialize a regex to match the G2O Data header and variables used during validation and to signify successful validation.

validation

sub recv_check_http_g2o {
        if (! re_g2o_auth_data.match(req.http.X-Akamai-G2O-Auth-Data)) {
                return (synth(400, "G2O no match"));
        }
 
        # backrefs:
        # 1 socket-ip
        # 2 client-ip
        # 3 time()
        # 4 unique-id
        # 5 nonce
 
        if (re_g2o_auth_data.backref(1, "") != "" + client.ip) {
                return (synth(400, "G2O IP mismatch"));
        }
 
        g2o_ts.set(std.time(real=std.real(re_g2o_auth_data.backref(3, ""), 0)));
 
        g2o_delta.set(now - g2o_ts.get());
        if (g2o_delta.get() > g2o_tolerance.get() ||
            g2o_delta.get() < 0s - g2o_tolerance.get()) {
                return (synth(400, "G2O time delta exceeded"));
        }
        if (re_g2o_auth_data.backref(5, "") == g2o_nonce.get()) {
                if (! blob.equal(
                    blob.decode(
                      decoding=BASE64,
                      encoded=req.http.X-Akamai-G2O-Auth-Sign),
                    g2o_1.hmac(
                      blob.decode(
                        encoded=req.http.X-Akamai-G2O-Auth-Data +
                          req.url)))) {
                        return (synth(400, "G2O Sign check"));
                }
                # at this point, we can believe it's really Akamai
                client_verified_akamai.set(true);
                set sess.timeout_idle = 6m;
        } else {
                return (synth(400, "G2O nonce unimplemented"));
        }
 
        unset req.http.X-Akamai-G2O-Auth-Data;
        unset req.http.X-Akamai-G2O-Auth-Sign;
        unset req.http.X-Akamai-G2O-Auth-Data-Query;
}

The VCL subroutine recv_check_http_g2o does the actual heavy lifting.

When we see a well formatted G2O Data header, we first check if the Akamai IP address matches the one we are talking to. Notice that this check might need adjustment if client.ip does not contain the actual peer's address, as for example with additional proxies in the path. client.ip will, however, be correct for use with a TLS offloader using the PROXY protocol to communicate with Varnish.

Next, we convert the Unix Timestamp from the Data header to a VCL TIME value and assign it to the variable g2o_ts, which we use to calculate the delta and check that against our tolerance.

Next, if the nonce field matches our configured id, we check if the signature is valid. The statement if (! blob.equal ... does a couple of things:

decode the Sign header into a BLOB
calculate the hmac of the Data header and the URL
compare the two BLOBs for equality

If this check succeeds, the signature is valid and we remember that fact in the client_verified_akamai variable for later use. Here we also demonstrate how we might want to relax some security/dos-related parameters like the session timeout.

Last, we remove the G2O headers in order to not mislead any backend services to rely on them.

To use the code, just call it with call recv_check_http_g2o; from vcl_recv . Then, at any point on the client side, if (client_verified_akamai.get(false)) can be used to carry out some operation only if we are talking to Akamai.

Note that we do not check the unique-id as saving and checking recently seen ids would drastically raise the computational cost for the check, in particular with a cluster of varnish-cache servers. By not checking the unique-id, we leave a window of opportunity to replay requests within the g2o_tolerance delta. On the other hand, due to the socket-ip check and with TLS in particular, launching replay attacks is particularly hard, rendering them rather irrelevant for all practical purposes.

Discussion of signed requests vs. CDN IP whitelisting

With IP whitelisting, we restrict access to an origin on the IP layer, which is the most efficient option, but does exactly that — make any other direct origin access impossible. In section The PROs and CONs of IP whitelist approaches we identified for which attack vectors IP whitelisting does and does not help and noted that only restricting access to all of the origin services will provide relevant additional protection.

Checking header signatures is less efficient (a connection needs to be accepted and HTTP headers processed), but the additional check is still lightweight (in the order of >> 100k req/s per CPU). Other means can and should be used to protect the origin from overload, most importantly rate limiting.

With signed requests, we get reliable authentication of CDN requests while still maintaining the highest degree of flexibility, allowing to use the CDN where it is most efficient and preferring direct requests where they are the best option.

While IP whitelisting might imply additional overhead in the CDN for routing requests through multiple hops, signed headers do not require any additional routing in the CDN.

IP whitelisting requires timely updates of changing CDN IP ranges (preferrably fully automatic), but signed headers do not require any regular maintenance (yet, signing secrets should be rotated regularly).

Summary

We have discussed two alternative approaches of authenticating requests from CDN nodes to an origin and their PROs and CONs.

IP whitelisting sacrifices flexibility for the most efficient filtering and might help mitigate some attack vectors.

Signed headers are a good alternative for sites with an efficient, resilient origin infrastructure to gain maximum flexibility.

We have shown the implementation of signed headers by example of Akamai G2O with validation in varnish-cache.

Acknowledgements

Thank you to Michael Weber of SPRING for helpful discussions and feedback. Jim Goroll created the diagrams and found an important aspect missing in an earlier version.