AB Testing Framework with Nginx and Varnish

AB Testing Framework

Problem Statement -

Perform AB Testing on product pages, AB testing is essentially an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better based on analysis. Variation which has performed better based on analysis, will be deployed on production.

In Traveltriangle infra, we are caching pages after the first hit returned from the server in varnish cache, so that next time request comes for the same url, page is returned from cache. We also have to handle this while doing AB experiments.

Tools Used in Traveltriangle -

In Traveltriangle, we are using GTM (Google Tag Manager), and VWO (Visual website optimizer) as AB testing tool. In GTM, we generally create a custom HTML tag that includes A/B testing script and the code for the variants. You need a trigger to fire the tag. This trigger can check — platform of request (Desktop, Mobile), Which pages to include script for, also including a script for a particular set of users etc, and the same is the concept for VWO.

Problems with Existing tools-

One of the biggest downsides of using Google Tag Manager for A/B Testing is that it still has a flickering effect on the pages that are loaded. That’s the biggest issue with running testing through the front-end. GTM does the work well except flickering on pages (because Code is loaded after the page has been loaded by GTM script), This flickering can be minimized if we load A/B testing script earlier in page load but can not be avoided.

Another one is, suppose we have 5 components in our home page and every component hits its own api’s. In one of the experiments, we don’t want to show one component, then VWO or GTM script can only hide component with flicker, it can not stop that component from hitting its api.

Solution to the problems -

To avoid flickering, we searched for many solutions, then we thought of building our own with the help of nginx and varnish. We read some blogs on A/B testing in nginx and A/B testing in varnish. We combined both solutions and tried to build our own. So if we are setting some experiment cookie at the web server itself and let the application server know which experiment it needs to run, then we can overcome this flicker, because page will be constructed after application server reads cookie.

Tools used for New Solution-

Nginx with nginx lua module, Lua module is used for embedding the power of Lua into Nginx HTTP Servers. Lua is a scripting language, given scripting powers to nginx.

Varnish.

Diagram of AB Testing Framework used in Traveltriangle

How we have used this setup -

Let’s suppose we have to run AB tests for our home page traveltriangle.com. Please refer attached screenshot -

We need to experiment on explore form, instead of 3 drop downs as — destination, duration and month, in our variation, we will only show destination drop down and explore button in our variation.

Like this -

Variation — 1 — Explore form with 3 fields. — to 30% of users.

Variation — 2 — Explore form with 1 fields. — to 70% of users.

Now we need to tell the front end which variant we want to display to the user (var1 or var2). In order to achieve this, if we set cookie’s — cookie_explore value to var1 for 30% of traffic and var2 for 70% of traffic, this cookie’s value will be read by front end and then serve right HTML for the version.

Since TravelTriangle is using nginx as its web server and node as application server (hits always comes to web server (Nginx) first and then routed to the application server (Node), we should set this cookie according to traffic percentage in Nginx itself. Now problem arrives -

how we set cookie value to var2–70% of times and var1–30% of times -

There is no random function and other in nginx configuration language, but we can use lua nginx module, which provides all standard functions to nginx. It gives power to nginx.

So the idea is, we are taking a nginx variable, let’s say — experiment_1 and set its value like below -

// This is lua code.

if ngx.var.experiment_1 == “not_set” then

math.randomseed(os.time())

local rand = math.random(1,10);

if rand < 4 then

ngx.var.experiment_1 = “true”;

else

ngx.var.experiment_1 = “false”;

end

end

This lua script sets variable experiment_1 value to var1 for 30% traffic and var2 for 70%. And we set can now set cookie as follows: -

Nginx code.

// For setting response cookie -

add_header Set-Cookie “experiment_1=$experiment_1; Path=/; Max-Age=864000”;

// For setting request cookie -

proxy_set_header Cookie “experiment_1=$experiment_1; Path=/; Max-Age=864000”;

Imp. We also want to show only one kind of variation to a user — for this we already have set a cookie experiment_1, again we need to read this cookie value and don’t set if this is already set.

map $experiment_1 $experiment_1 {

not_set not_set;

true true;

false false;

}

So as of now, our node server is getting information in cookie value about which variation to show.

Since we cache our pages on varnish, so for home page, we need to have 2 cache entries, In Varnish, this is possible by setting a vary header by specifying a header with request and ask varnish to vary caches based on this header.

Varnish Code -

In subroutine — sub vcl_backend_response

if(!beresp.http.Vary) {

set beresp.http.Vary = “experiment_1”;

}

else {

/************** START — Experiment Purpose — Vary on specific headers ************/

if(beresp.http.Vary !~ “experiment_1”) {

set beresp.http.Vary = beresp.http.Vary + “, experiment_1”;

}

/************** END — Experiment Purpose — Vary on specific headers ************/

}

With this code, Varnish will cache 2 variants of home page, and serve right content upon requests.

Real time experiments with Traveltriangle’s ab testing framework -

In Traveltriangle, we are running many experiments with the help of AB testing framework — please refer attached screenshot We wanted to have different lead forms on the same page, to see which lead form can give more conversions, This experiment will be running on the following URL -

https://traveltriangle.com/mkt/kerala-tour-packages

With cookie_1(cookie name) value as “true”, we want to show new lead form version, and cookie_1(cookie name) value as “false”, we want to show old lead form version.

So this lead form is embedded to HTML through a script. Let’s say — lead_form_v1.js and lead_form_v2.js

We include these javascripts based on reading cookie value of cookie_1, if value is true, then we embed lead_form_v1.js else lead_form_v2.js in application server.

New Lead form when cookie_1 = true

Old Lead form when cookie_1 = false

Conclusion -

If you have Nginx for request routing and Varnish for request caching, then this solution is perfect for AB testing.

Nginx configuration is a very sensitive configuration, if you have wrongly added a cookie header to a request, then sometimes nginx does not forward headers to the next request.

A example of the same is as follow -

Suppose we have 3 experiments to run — then — in all 3 conditions, we need to add header for all of them, otherwise header would not be passed.

if ($set_cookie ~ ‘true’) {

add_header Set-Cookie ‘param1=$param1; Path=/; Max-Age=864000’;

}

if ($set_exp_cookie ~ ‘true’) {

add_header Set-Cookie ‘param1=$param1; Path=/; Max-Age=864000’;

add_header Set-Cookie ‘param2=$param2; Path=/; Max-Age=-1’;

}

if ($cbu_exp_exist ~ ‘false’) {

add_header Set-Cookie ‘param1=$param1; Path=/; Max-Age=864000’;

add_header Set-Cookie ‘param2=$param2; Path=/; Max-Age=-1’;

add_header Set-Cookie ‘param3=$param3; Path=/; Max-Age= 1209600’;

}

A lot of interesting things happening @ TravelTriangle. Join us in solving these challenging problems and creating a world-class holiday B2C and B2B ecosystem; email at lead_on@traveltriangle.com
If you know anyone in your network who might be interested in solving these problems, do share this article.
If you like the article, do like and/or clap so that others might stumble upon this article.