Templating your content delivery network using Fastly: one service, many websites
Here at USA TODAY NETWORK, we maintain 132 websites through a single Fastly service, allowing network-wide changes go out in the same amount of time as a change to a single website with minimal effort on our side.
We use Terraform to manage our Fastly services and VCL. Thanks to the way we template out our code, our Terraform and VCL files do not get unmanageably long. We use a bit of Golang code we wrote, dubbed our “templater”, to generate VCL and Terraform documents for us based on JSON files. This means we never directly edit our custom VCL for this service, which spans out across eight VCL files and is quite lengthy.
Templater starts by loading up all of our required domains from a JSON file. Currently, our service has 298 domains. This JSON file has a format:
[
{
"hostname": "ithacajournal",
"tld": "com"
},
{
"hostname": "pressconnects",
"tld": "com"
}
]There are a number of options that can be provided: the hostname, the top-level domain (TLD) or what experience you want the user to have. Here you would specify if the site should use the new web platform we are rolling out on mobile or desktop, and in just staging or also production. This allows for extremely easy testing and cutovers for our new web platform.
Templater then loads and creates files based on Terraform, VCL and Golang templates. In order to create the Terraform file, we iterate on the values provided in our JSON files. For example:
{{range.}}
##### {{.String}} config #####
domain {
name = "${terraform.env == "production" ? "{{.String}}" : format("%s-%s", terraform.env, "{{.String}}")}"
} domain {
name = "${terraform.env == "production" ? "www.{{.String}}" : format("%s-%s", terraform.env, "www.{{.String}}")}"
}
{{end}}
This code creates a chunk of Terraform defining each necessary domain. We have two domains for each single domain in Fastly, one prefixed with www and one not. The final Terraform code would look like this:
##### ithacajournal.com config #####
domain {
name = "${terraform.env == "production" ? "ithacajournal.com" : format("%s-%s", terraform.env, "ithacajournal.com")}"
}domain {
name = "${terraform.env == "production" ? "www.ithacajournal.com" : format("%s-%s", terraform.env, "www.ithacajournal.com")}"
}
We also template out and generate our VCL files. This is how we would create a massive table:
table some_useful_values {
{{range .}}{{if .ValueExists }}"{{.String}}": "1",
{{end}}{{end}}
}This spits out some VCL with a table in it that looks like this:
table some_useful_values {
"detroitnews.com": "1",
"freep.com": "1",
"rgj.com": "1",
"courierpostonline.com": "1",
"cincinnati.com": "1",
"news-press.com": "1",
"floridatoday.com": "1",
"theadvertiser.com": "1",
...
}We even template out some of our tests:
func TestProperties(t *testing.T) {
t.Parallel()
var tests = []SomeTestTable{
{{range .}}{{if .TrueFalseValue }}
{
description: "www.{{.String}} succeeds some test",
expectedStatusCode: http.StatusOK,
regexMatches: []string{`regexToMatch`},
expectedRegexMatchResult: true,
doNotPurge: true,
baseDomain: "www.{{.String}}",
userAgent: UserAgentGoogleBotDesktop,
URI: "/",
},
{{end}}{{end}}
} TestBody(tests, t)}
Resulting in tests per property for everything defined in that template:
func TestProperties(t *testing.T) {
t.Parallel()
var tests = []SomeTestTable{
{
description: "www.ithacajournal.com succeeds some test",
expectedStatusCode: http.StatusOK,
regexMatches: []string{`regexToMatch`},
expectedRegexMatchResult: true,
doNotPurge: true,
baseDomain: "www.ithacajournal.com",
userAgent: UserAgentGoogleBotDesktop,
URI: "/",
},
}TestBody(tests, t)}
The files generated from these templates are what gets uploaded to our Fastly service via Terraform. The final structure of our code contains the following things:
- AB testing
- Device detect
- Referrer detect
- Browser detect
- OS detect
- Bot handling
- Request logging
- Extensive surrogate keys
- Redirects
- Forcing HTTPS
- 404 pages
- Platform specific whitelist and blacklist
While not a fully comprehensive list, this demonstrates how extensive the VCL and Terraform we’re generating is. And thanks to the way this is designed, we’ve been able to slowly roll out an entirely new web platform, releasing and migrating traffic completely through Fastly in a seamless way we wouldn’t have been able to do before.
Making sure we don’t make a mistake
So generating and editing this code is easy — how do we debug and test it when our changes are liable to have such a large impact? To assist debugging such a large codebase, we developed our own custom header. When set, this header returns information about what path the request took through our VCL, including redirects, backends, whether or not it’s part of our new web experience — basically anything you could think of that might help to debug a pesky request. This header is also a fundamental part of our testing pipeline.
Rather than touch on the technicalities of how we write our tests, let’s focus on the content. If you want more information on how we test our services on every pull request, check out this blog post on testing Fastly services on every PR.
We have two sets of tests that are templated and generated by Templater, our whitelist testing and our property testing. Our property tests simply ensure each domain is going through the correct backend (we use our custom header to determine this) and getting the right version of the site. Our whitelist testing ensures that anything whitelisted for our new web platform is actually getting our new web platform. The rest of our tests are written by hand and ensure that every piece of VCL written has testing. This includes testing our redirects, bot rules, surrogate keys, SSL enforcement, TTLs, unsupported browsers, device detection and the like. In addition to this, we have a separate suite of tests for QA that test desktop, sitemap and mobile.
Setting up our service this way allows for simple changes and thorough testing of the cache for our entire USA TODAY NETWORK. Not only are changes quick, they’re version controlled and deployed immediately.
