How we are recreating old incidents to ensure our engineers are prepared for the unexpected

A fireman putting out a practice fire made of wood with a foam spray.
A fireman putting out a practice fire made of wood with a foam spray.
Practice makes perfect, ideally in a controlled environment. Photo by Hush Naidoo on Unsplash.

Within the Customer Products group at the FT (we work on and the FT mobile apps) we’ve faced an interesting situation this year, something for us to be proud of. We have experienced only a small number of incidents that have significantly impacted our products, this is in stark contrast to previous years where out-of-hours calls to engineers were much more frequent.

Alongside this, our group’s tech strategy includes a whole area of focus dedicated to “making our out-of-hours process sustainable”. With the risk of us being unprepared to handle our next major incident we set out to plan…

Photos from a cold adventure with Ray Mears.

This April I spent ten days away in Finland, on a course run by Woodlore that taught us the skills needed to travel Earth’s Taiga biome.

The Taiga, also known as the Boreal Forest, spans the Northern Hemisphere and is the second largest biome on Earth. It sits just below the Arctic tundra and lies within the Arctic Circle.

Based in the very North of Finland, we were staying around Lake Menesjärvi, often skiing over it to campsites near the shores.

Spending several nights out in the snow under the Milky Way there were some amazing views, and if you…

Our road to enabling HSTS, inspired by the Yelp story.

“A turquoise chainlink fence with a padlock in Greece” by Jason Blackeye on Unsplash

Since October 2016, when we launched the new site, has only been available over https://.

However, we still have many insecure links pointing to our site, leaving users at risk from session hijacking and protocol downgrade attacks.

Over the past few months we’ve implemented the HTTP Strict Transport Security (HSTS) specification on, to ensure that our users will only ever talk to us over a secure connection.

This is the story of how we got there, and the hurdles we hit.

What is HSTS?

HSTS is a declaration websites can make using HTTP response headers, that tells browsers to only use…

Making a request to the Financial Times.

The Financial Times front page.

This is an overview of how the Financial Times serves requests to Starting with our domains, going all the way down to our Heroku applications, and through everything in between.

Table of Contents

  1. Domain Name System
  2. Content Delivery Network
  3. Preflight
  4. Router
  5. Service Registry
  6. Applications
  7. Elasticsearch
  8. The End Result

Domain Name System (DNS)

What is Brotli, and why should I care?

I’m hotly for Brotli 🔥

Brotli is a new-ish lossless compression algorithm developed by Google.

Brotli was initially developed to decrease the size of transmissions of WOFF2 web fonts, and in that context was a continuation of the development of zopfli, which is a zlib-compatible implementation of the standard gzip and deflate specifications.

So it’s very web oriented.

Serverless redirects for legacy domains using Fastly, Terraform, and CircleCI.

The final CircleCI 2.0 workflow, working a treat 🙌

Previously at the Financial Times we’ve made use of the Dyn HTTP redirect service to send traffic from domains and subdomains to, which has been working just fine for our legacy domains that never used TLS.

However, there has been a number of recent tasks to redirect subdomains with support for TLS, and so far several of these domains have ended up under our Fastly service, a scary thought given the complexity the configuration (over 5,000 lines of VCL).

What I’ve been looking to work on recently was a continuous delivery pipeline composed of Terraform and CircleCI 2.0…

Serverless log file analysis with web scale.

What is Fastly?

Fastly is a content delivery network, powered by the Varnish HTTP Cache. It enables the Financial Times to deliver globally in under a second.

What is Amazon Athena?

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.

Which means, for a practical example, if you have logs stored in S3 in a typical log format, one request per line. [2017-08-30 00:01:43.000] "GET / HTTP/1.1" 200 12846 TLSv1.2 HIT [2017-08-30 00:01:43.000] "GET /robots.txt HTTP/1.1" 200 68 TLSv1.2 HIT [2017-08-30 00:01:43.000] "GET /not-a-url HTTP/1.1" 404 0 TLSv1.2 MISS

We can use…

Samuel Parkinson

Principal Engineer at the Financial Times.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store