5 things to take into account when running your A/B testing experiments

Make every detail perfect and limit the number details to perfect.

– Jack Dorsey, Square

If you are -like me- somebody passionate about building digital products, it’s likely that you’ve used A/B testing techniques to improve the user experience in many projects.

In this short article, I’m going to highlight five aspects that I always take into account when doing my A/B testing experiments.

1. Start by defining the A/B testing hypothesis

The first step of the A/B testing process must be defining the experiment hypothesis. It’s important that this is well set at the very beginning of the process as it helps define further details of the experiment like user segments.

Ideally, any changes to your app must be related to one (or more) of your product key performance indicators, KPI(s); this approach helps you define ‘success’ and gives you the framework to analyse if the changes were worth the effort.

I have a ver simple way to create the A/B testing hypothesis just by following this structure:

Making a variation of [element(s) to change] will have a positive/negative on [relevant KPI].

2. Don’t change everything at once.

Make sure each variation contains the minimum possible number of changes in your user experience that allow you to move the previously selected KPI.

For example, if you try to increase the trial conversion rate, start by changing the button your users click to subscribe (different position, style highlighted, more direct call-to-action, etc.) as opposite of changing the whole subscription flow completely.

The goal is to isolate the changes so you minimise the noise and are able to test what truly has a real impact.

3. After defining the variations, pick the right time frame and audience.

How long is the experiment going to run? Depending on the area you decide to test, it could go from a short period of time to multiple weeks. Make sure you set a time frame that allows you to collect the relevant amount of information to generate actionable insights.

Is this experiment relevant to all your users? If, that’s the case, randomly displaying the variation to half of your users should be enough. However, if the answer is more complex, you probably want to take extra time and define the pertinent segment(s) of users as they are the ones that ultimately are going to move your KPI.

4. For your test, use existing tools whenever possible.

Whether you want to run your A/B test across all users of your app or during a usability testing session with a few of them, try to use existing tools as much as possible. They normally work out-of-the-box and avoid you building custom A/B systems.

At the end of the day, that’s your not core business (unless your product is an A/B testing tool!).

5. Once finished, take the time to analyse the results before making decisions.

A/B tests are data-driven experiments so before jumping on decisions, spend the time to analyse the results. Make sure the quality of the data was good enough and that there weren’t any external factors making a positive/negative influence on your users’ behaviour.