Multi-Variate Testing on the BBC Sport App

Building a better app for the next generation

The year 2012 is significant for many reasons:

1. It was the year the world was supposed to end
2. London had the Olympics
3. The Queen celebrated her Diamond Jubilee

But most importantly, it was the year that BBC Sport started working on its mobile presence. It all began with the BBC Olympics app. Five points if you can work out what it was for…

The following year BBC Sport was introduced as its own app to cover all the other sporting events that were happening, not just the Olympics. However, in order to satisfy the need quickly and efficiently, the app ended up being a web-wrapped version of the BBC Sport website. Which, seven years ago, wasn’t so bad. But now we want all the newest, shiniest features in our apps and we have to step up or get left behind.

As a team we are working hard to distinguish our mobile presence from our web presence. But, as a product with an audience of approx 3 million, we have to be extremely careful when making big changes.

Which bring us to Multi-Variate Testing (MVT)

MVT is a method of rolling out new features to a select audience to see how well they are received. The “Multi-Variate” part is what sets it apart from A/B testing as it means you can have as many different variations of a new feature as you like, rather than being limited to just 2, although, it is recommended to keep this between 3 and 4 variations per experiment.

At the BBC we implement most of our Multi-Variate tests using Optimizely as it has proven extremely useful at helping us understand our users.

Fast-forward to 2019, and our user base has changed dramatically from 2012. The needs of sports fans have changed, how they consume content has changed and what they want from the BBC has changed. How do we handle that? How do we address the needs of every type of user? But how do we keep a hold of our old users who like the app the way it is? Bury our heads in the sand? Panic? No (well yes, but not as the main solution).

You guessed it, Multi-Variate Testing!

Multi-Variate Testing Example

Over the last few months we have gathered lots of ideas to make the app better, some that are aimed at specific audience segments and some that are more generic. Eventually, we had one idea ready to go. The designs were done and the developers were ready to start working on it. But… we needed more evidence to say that our users would actually appreciate the new feature. We needed concrete proof that our users would continue to enjoy using the app with the new feature.

To do this we used a service named Optimizely.

MVT is great for experimentation as it allows you to define an audience, e.g. UK users or worldwide users, and then provide a variation for everyone in that audience. This variation will never change for a user as it is linked to their device ID, which means we don’t need to access any personal data about the users in the experiment. Furthermore, we can limit the reach of the experiment so you can define if you only want 10% of the audience to see the experiment or everyone. This is useful for limiting the damage of a poorly received experiment.

We came up with three variations of a new feature and defined the conditions for each variation. We then created a new experiment in the dashboard. This detailed:

• The experiment name, e.g. test-experiment-one
• The different variation names, e.g. variant-a, variant-b, variant-c
• The audience who were able to be in the experiment, e.g. 20% of UK users who have signed in to the app

Once we had this information we could then use the variation names to control which variation the users saw. When a user opened the app we would assign them a variation and then, wherever needed in the app, we could ask our experiment service what variation the user was in and change the view accordingly.

We also had the ability to report metrics on what the users were doing. For example, we could report that a user followed a new sport in My Sport. Then, in the dashboard we can see which variant enabled the most people to follow a new sport. This was incredibly useful for our experiment and gave us real-time feedback on how well the experiment was performing.

Example dashboard from Optimizely

The experiment ran for four weeks to gain real statistical significance, ie we could say that one variant was better than the other in a meeting and have reliable numbers to back that up. However, after two weeks the experiment running on Android had to be turned off due to a crash (on our side) affecting lots of users. Despite that, we managed to keep the iOS experiment running for the full four weeks and the results were similar enough that we could use both results and be confident in choosing a winning variation.

For this experiment we needed to implement Optimizely from scratch which added some complexity to getting the experiment running.

Diagram of our Optimizely Implementation

Once we had created a service to talk to the MVT provider within both our codebases (iOS and Android), we had all the pieces in place to be able to kick off the experiment. In total, it took approximately six weeks to build the variants and integrate the newest version of the SDK. Then a further four weeks for the experiment to run.

So in ten weeks we had a concrete idea of whether users would like our new feature or not. In the past, this wouldn’t have taken weeks, instead it would have been months to decide. Using MVT in our app has definitely made it easier to gain insights into the audience and reduce the fallout of us releasing big bang changes to the public.

Due to the success of the first experiment, we now intend to use MVT to gauge the public reaction to all of our big changes. An insight into a user’s mind is priceless for app development.

Ultimately, we wouldn’t be able to make a truly user-first sports app without finding out firsthand what it is that users want from the app. MVT has allowed us to do this without having to intervene and ask users about their needs. Therefore, we are now able to discover what our users want and trial it out before we push it to the entire user base. We recommend MVT testing to all app developers, it will give you an insight that you won’t get from a survey!