Supercharge Your Product With A/B Testing

Use data-driven testing techniques to create an application or website that your users can’t live without

Todd Runham
Aug 17 · 5 min read

When building a tech product, there are a lot of unknowns you have to deal with.

We’re going to be focusing on the last question, and how to answer it using various forms of A/B testing. This will then enable you to create product growth that is guided by constant user feedback. It also means you can design and develop based on the behaviour of users, rather than attempting to predict their expectations.

What is A/B testing?

A/B testing is a straightforward concept. You take a section of your product, and you modify it. It may be an individual element like a button or a larger entity such as an entire page. You then send a portion of your traffic (usually 50%) to the new version of this section, known as the variant. The rest of the traffic will continue to see the original, known as the control.

Throughout a predetermined period of time, you will monitor engagement with both versions. If the modification generates a positive reaction, you know you have introduced a successful change.

How can it help your product?

Improving a product at a tech company receives a large amount of focus, yet is often done incorrectly.

A considerable amount of time and effort is spent on pushing as many features out the door as possible. Usually, because of the misconception that more features will always mean more users. However, this is simply not true.

Polluting the UI with too many components that are rarely used is counterproductive. Instead, new changes should be treated as experiments that can have the following outcomes:

  • If the feedback does not indicate any value at all — there is no need for it.
  • If the feedback is inconclusive — it’s worth investigating how to make it more appealing to the end-user.
  • If the feedback is positive — finalise the changes and move on to the next idea. Alternatively, you could improve upon it even further to maximise success.

When developing a new experiment, it’s best to build a minimum viable product (MVP). An MVP in this context is where you build enough of the feature to satisfy basic requirements. If users do not respond well to the variant, at least you haven’t excessively leveraged development cost. As a consequence, your development cycle time will not only be a lot faster but more efficient as well, as waste is being eliminated.

So we’ve established how you should introduce alterations to your product. Now we need to talk about metrics, and this is where A/B testing comes into play.

With A/B testing, you have a point of reference by retaining the original version. You can compare metrics sourced from this to metrics sourced from the new variant. They can be anything from traffic to conversion rates, or on a more granular level — button clicks.

These metrics will then be a great indication of which path should be taken, as they will exhibit better engagement in either the “A” path or the “B” path.

For example — Increasing the size of a CTA that links to a page in your application that receives little to no traffic. By comparing the page views of the users across both the variant and control buckets, you will be able to determine if increasing the visibility of the CTA has helped increase visits to the unfrequented page or not.

Advanced A/B testing

Once you have the hang of two-track experimentation, you can start looking into multi-track. More commonly known as multivariate or A/B/n testing.

Multivariate testing is more difficult as you have various combinations of changes you need to analyse and keep track of.

On the other hand, it does allow you to run concurrent experiments in the same area of your product, such as three variations of a promotion. This can further decrease cycle time and reduce waste as you don’t have to run these tests sequentially. It also gives you a lot of space to be creative.

There are a few ways to implement these methods. Larger companies may build an in-house solution. Google offers a free platform that has gained some traction recently and is a great choice for startups. A popular alternative is Optimizely, which is an excellent option for more established businesses that are willing to pay a third party for this functionality.

Whichever solution you lean towards, being able to visualise which improvements users favour means you can be confident that you are in fact introducing the right features.

Some best practices to end with

Stabilise — Remember that if the experiment is successful, it should be evolved into a full, stable feature. This is often forgotten, and before you know it, you’ll not only have a codebase that developers struggle to work with but an incomplete product as well.

Choose when to utilise A/B testing carefully — These tests need a reasonable amount of data to produce a reliable conclusion. If you are just launching your product, you may not be able to source the required amount of data, and test results will be unreliable.

Be wary of overlap — Running too many experiments in the same space can devalue your data. It’s imperative you know what is creating success, and what isn’t. While multivariate tests can alleviate this, it’s best to remain focused and keep colocation to a minimum.

No amendments — You will want to wait until the end of the experiment before making amendments. This is so your final data will represent a fair comparison.

Gousto Engineering & Data Science

Gousto Engineering Blog

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store