AB tests behind a CDN

One of the benefits of the dynamism of the web and web development is the speed at which we can deploy new features, and the flexibility in how we launch them. At Tictail, a marketplace and e-commerce platform, we constantly run tests where a small group of visitors are served new features or new content. We monitor how features are used, and if treatment groups perform better or worse than their control groups.

We serve tictail.com via Fastly, our CDN provider. This allows us to quickly serve cached versions of tictail.com all around the world from Fastly’s edge nodes. Fastly uses Varnish. The Varnish Configuration Language, together with Fastly’s extensions to it, gives us the ability to completely customize the routing and caching for tictail.com, directly at the edge nodes.

We love Fastly, and the speed and flexibility it provides. However, it introduces a few new and interesting challenges.

The problem

The user Buffy, a potential shopper who has never visited Tictail before, directs her browser to tictail.com. The initial request to load the page goes to Fastly, which now ideally would serve a cached version of the content.

Screenshot of tictail.com, taken 2016–12–26.

At the same time, we’re running an A/B test to see if having the top bar yellow instead of white converts better. 50% of users are shown a yellow top bar, and 50% are shown the usual white top bar. The backend must serve the response so that no user will ever see a flicker of yellow that changes to white, or white that changes to yellow (the flash-of-unstyled-content, or really, flash-of-incorrect-content). The backend must know the A/B test group affiliation of the user to do this.

How does Fastly know to cache both a yellow and white version of the content? How does Fastly know which version to serve to Buffy?

Our solution

We’ve come to a solution that works, and in hindsight is rather obvious: Fastly must place users into the control and treatment groups. We can configure Fastly to do this without even knowing which tests are running in the origin backend.

Steps:

1. Fastly generates a random sequence S of As and Bs of length N for all requests. If N = 2, the possible sequences S are “AA”, “AB”, “BA”, and “BB”.
2. Fastly sets the header X-Tt-Backend-AB: S and adds/updates the Cookie header to include tt_bab=S (short for Tictail backend A/B group).
3. The origin backend can now look at the X-Tt-Backend-AB request header, or the tt_bab cookie, and use that information to determine test group affiliation. For example, one experiment E1 could use the first letter of the sequence to determine if the user is in control (A) or treatment (B). Another backend experiment E2 could use the second letter, and so on. E1 and E2 would now have 50% of users in each group. E1 and E2 would also be independent. If we want smaller groups for some experiments we could look at multiple letters in the sequence, or use other client-provided request properties that the backend response already varies on, such as the Accept-Language request header. If we want to run more tests concurrently we’d either need to increase N, or accept that the tests are dependent. The relative size of how large or small the treatment groups can be ranges from 1/2 to 1/(2^N).
4. The backend returns the variant of the content for the specific test group. The backend also returns the response header Vary: X-Tt-Backend-AB (alongside other Vary values), telling Fastly to include the sequence S in the cache key.
5. Fastly makes sure to include Set-Cookie: tt_bab=S; Expires=<date_far_in_the_future> Path=/; in the response, setting a cookie for the user (only if the user doesn’t already have the cookie). That way, the user will keep its group affiliation for upcoming requests.

If the user already had the tt_bab cookie, step 1 will be skipped. Fastly will now try to serve a cached version of the resource, dependent on the cookie value. If none is found, Fastly continues from step 2. Even if step 1 is executed, such as for our user Buffy, cached content could still be served, if such content exists in the cache for Buffy’s value of S.

If the origin backend has no active backend tests, it can skip step 3 and 4. The sequence S will then not be included in the cache key, and the cache key space will be smaller.

When sequence S is part of the cache key, the cache key space will be increased by a factor of 2^N.

The config

The Fastly VCL we use looks something like this:

This setup is not only useful for running A/B tests. At Tictail we deploy new code and features continuously, tens of times per day. Not all features are available to all users immediately; we often deploy to staff first, for an internal test. When we’re satisfied it works and nothing is broken, we enable it for 25% of English-speaking users, and monitor how the feature is used. If everything looks good, we proceed with translations and roll out to 100%.

Conclusion: By assigning A/B test group affiliation directly in the CDN we gain the best of two worlds: The performance of a CDN, and explicit and full control over how we render and serve our content.