Running A/B Testing Using Google Optimize
Around 1 year ago, we decided to develop a new version (let's call it version 3) for our website to replace our old one (let call it V2). The reasons,
- Our infrastructure at that time could no longer tolerate the barrage traffic of users.
- Our old version was pretty much focused on the speed of the deployment back then and now the technical debt accumulated started to manifest itself where certain features of our product constantly break here and there.
- There were several new UI, UX flow, and tech stacks that we would like to apply. So developing a new product from scratch sound like a reasonable decision comparing to refactoring our old code.
But the next question will be How can we know that our new website (V3) will outperform our old one (V2)?
Enter Google Optimize.
So what is Optimize, It is a tool developed by Google that allows you to test variants of web pages and see how they perform against an objective that you specify. Optimize monitors the results of your experiment and tells you which variant is the leader.
I will not cover how to create Optimize Container here, as there are plenty of resources that you can find online (like here). My focus on this writing just to share our experience in term of:
- What kind of experimentation we did?
- How did we create the test?
- What performance metrics that we look for the A/B testing
Type of A/B testing experiments in Optimize
So once you create your container. Just click ‘Create Experience’ at the top right corner of your Optimize Container. And you will see multiple selections of experiments that you can conduct on your website
Let's gone through the 3 main experiences one by one
A/B tests
An A/B test, sometimes called an A/B/n test, is a randomized experiment using two or more variants of the same web page (A and B). Variant A is the original.
Variants B through n each contain one or more elements that are modified from the original (for example, a different colored call-to-action button, different copyright between 2 pages). In some experiments, variant B may be a completely different version of a web page.
Each variant is served at similar times so that its performance can be observed and measured independent of other external factors. You can also limit your experiment to a specific audience with targeting. Optimize captures the performance data for each variant against an objective and identifies the leader.
Example
An A/B test of two variants of a product page (A and B). Variant A has a blue photo, while variant B has a red photo.
Multivariate tests (MVT)
A multivariate test (MVT) tests variants of two or more elements simultaneously to see which combination creates the best outcome. Instead of showing which page variant is most effective (as in an A/B test), MVT identifies the most effective variant of each element as well as analyzing the interactions between those elements. MVT tests are useful for optimizing multiple aspects of a landing page, for example.
Example
You can test two headlines (original “H1” and variant “H2”) and three hero images (original “A” and variants “B” and “C”) simultaneously, which would produce:
- 2 sections (the headline and the images), with
- 3 and 2 variants (respectively), for a total of
- 6 combinations (the number of variants multiplied by each other, or 3 x 2)
Redirect tests
A redirect test, (a.k.a. split URL test), is a type of A/B test that allows you to test separate web pages against each other. In redirect tests variants are identified by URL or path instead of an element(s) on the page. Redirect tests are useful when you want to test two very different landing pages, or a complete redesign of a page.
Example
Test two different landing pages, with different URLs:
- Original — www.example.com/landing1
- Variant — www.example.com/landing2
Test a redesigned page hosted on a subdomain:
- Original — www.example.com
- Variant — new.example.com
Our Experiments
Back to our problem statement, we want to test between 2 versions of our website (V2 vs V3) and for that, we will be using Redirect Test in Optimize. The moment the users land on our landing page. We will set a 50/50 split. Which means that 50% of the users will go to the old version while the remaining 50% of users will go to the new version. Each web page in both versions will have totally different URL as we want to A/B test 2 different website flow (not just the first page).
After we choose ‘Redirect Test’, under Targeting and Variants, click Add Variant. Basically here is where we declare the old version and the new version we would like to test
After finish declaring all the variants, we can then adjust the traffic we want to split for each version. For our case, we would like to split 50/50 traffic amongst Original(V2) and Version3 (new)
Page targeting
Page targeting is basically where or when the experimentation will run on our website. For us, we want the experiment to run after customers reach the second page (which is the booking page) from our landing page.
Audience targeting
We can filer the user segment for our experimentation under Audience targeting. we can set if we only want users that use specific devices (Desktop, Mobile and Tablet) to access our website, or based on their behaviors (New user or repeated user) to the point of Geography (City or state they are coming from). For us, we just want to test users that uses Desktop.
Objectives
Objectives are metrics or activities that your variants are measured against. Your experiment objective determines your experiment’s status and which variant is the leader.
Optimize offers at least 3 objective settings under the free tier, 1 primary objective, and 2 additional objectives. Our primary objective for this experiment was that we want to see Transactions from these 2 different versions, and under additional objectives, we want to measure Pageview and Session Duration.
Note: you have to enable e-commerce tracking in Google Analytic first to get the transaction objective inside Optimize. Otherwise, you can create a custom objective that measures customers that reaches your transaction page(For us, our transaction page will be /thankyou).
Settings
Now, to make sure your Optimize is installed properly, you can check the installation under Setting. The most common issue that happens during Optimize installation involves ‘anti-flickering snippet’ which you can overcome using this guide
Once you finish setting up everything, you could start the experiment by clicking ‘Start ‘ at the top right of your Experience Page.
Results
For the result to become statically significant. Google advises us to run the experience at least for 2 weeks. Optimize uses Bayesian inference to generate its reports. Bayesian inference is a method of statistical analysis that allows Optimize to continually refine results as more data is gathered. While computationally involved and expensive, Bayesian inference offers four key benefits compared to more traditional approaches:
- Allows Optimize to compute probabilities directly to better answer questions like, “What’s the probability that the new variant is better than the original?”
- Avoids a common misunderstanding that p-values are the same as probabilities, allowing Optimize to provide relevant data that’s actionable.
- Allows Optimize to determine the probability of any one variant to be the best overall, without the problems associated with hypothesis-testing approaches.
- Allows you to end an experiment as soon as Optimize finds there isn’t much more to learn by continuing to run the experiment.
Here is the result after we run our first experimentation.
Our initial result shows our V3 conversion is much lower compared to our original version. After investigation, we found out that there was a bug that causes our v3 Booking page to has a high page load time. After some minor fixes, we are able to optimize our V3 page load speed and run again the experiment.
Based on the latest result, we are able to improve our conversion rate to 20% more compared to the old version. I am reaching the end of my articles, thank you for reading.
