A Quick Lesson In A/B Testing
What is AB Testing? AB Testing compares 2 versions of a webpage with real users to determine which page works better. For example, on a landing page, you can trial different colours for your main Call to Action to see which colour gives the best conversion rate.
On most websites, changes are made to improve a metric. These metrics differ widely across different industries, but almost all websites have some key metrics. Here are a few examples:
- E-commerce: Revenue is a key metrics as this is where your companies money is probably coming from. If your product pages aren’t encouraging people to buy your products, then you’re missing out on sales revenue.
- News: Article read is an important metric. It doesn’t matter how good your journalists is or how many articles they write, if the users don’t like the layout of the page then they will go elsewhere.
- Social Media: Active users is also an important metric when it comes to social media. The more active users you have, the more content is being posted to be engage with. Imagine how boring Facebook or Twitter would be if there weren’t any other users actively using those platforms.
- B2B: The time a user spends per action when it comes to A/B testing. If your site is too time-consuming and difficult to use, then business will take their business elsewhere. For example, If it takes 20 button presses and 10 minutes to approve a colleagues holiday time, that business will move to a different platform which only takes 2 button presses and 10 seconds to do the same action.
Before we talk more about A/B Testing it’s important to understand metrics. Metrics are quantitative, as you can measure the changes over time. From the above examples, `revenue` is a monetary value, `Articles read` and `Active users` and numbers, while ‘Time per action’ is how long it takes a user to complete a given task.
Changes on your website will directly impact these metrics. If the layout of your news articles isn’t easy to follow then users will get frustrated and get their information elsewhere.
How do we know what changes will improve these metrics? We do experiments. We can have 2 versions of the same page, so half of the users visit one version of the page, while the other half visit the second version. Then we compare the performance of these 2 pages to see which had the best performance.
An example could be that you have a landing page where the main call to action is a blue button, and this landing page has a 4% conversion rate. Your A/B Test could be to create a second version of the page where the main call to action is yellow. With both versions of the page running at the same time, you can see how they perform against each other. If the blue button keeps its 4% conversion rate and the yellow button has a 5% conversion rate then you’d keep the version with the yellow button and remove the blue button; if the yellow button instead had a 3% conversion rate, you’d keep the version of the page with the blue button.
When we do A/B Testing it’s important to only make one change at a time. If you make a version of a page with 2 changes then you’re not going to know which change is causing the increase or decrease in your metrics. You could change the colour of your primary call to action and you could change the text in your heading, but if the metrics decrease you won’t know which change caused this: Maybe the button was a positive change in your metrics, but the text change caused an overall decrease in that metric.
There are times when you do want to test more fundamental changes to the design, in which case you’d do Split Testing. Similar to A/B Testing, you have 2 different versions of a page and your users see either version A or B, but the versions are testing 2 fundamentally different designs.
There are many things you can test with A/B Testing that will improve metrics:
- Text changes: such as the heading of a page
- Colour changes: button or text colours
- Style alterations: such as curved VS square buttons
- Layout of visual elements: different ways of displaying forms
- And much much more
A great thing about A/B testing: There are a lot of tools out there that do a lot of the testing for you. They let you change the appearance of your site via an editor, they run the experiment, and they show the results. For most platforms the only development time required is to insert some Javascript into your website that allows the experiments to run.
To summarise: A/B Testing is a very useful method to improve the performance of your site without having to get your development team to make lots of little changes. They can focus on “the big stuff”, while you focus on optimising the features you already have.
About the author:
Ed is a product manager for the e-commerce platform Flubit. Initially working on the B2B products, he led the rebuild of the Weflubit merchant site and has overseen its continuing development. More recently he has also become involved in the consumer Flubit site and is helping to make the site more user focused. Away from the office, he is a long-term volunteer for Mind The Product and can frequently be found helping out at their events and meetups.
