Split Testing (A/B Testing) with Swift

Jul 4, 2018 · 7 min read

Nowadays there are a million apps that solve users’ needs, so it is becoming hard to create something new or be better than market competitors. This has led to a situation where a lot of companies and startups are running experiments to find out which features are actually improving their product and which they should throw away.

One of the main tools to run such experiments is Split Testing (or A/B Testing), and so this article will cover this instrument and gives an example how to carry out a Swifty implementation.

The whole demo project is available via this link: https://github.com/azimin/SplitTestingArticle

If you already know what A/B Testing is, you can jump into the code part.

A brief introduction to Split Testing

Split Testing or A/B Testing (which is less applicable in my opinion because usually you have more than 2 groups) is an instrument to check different versions of something in different user groups to find out which version performs better.

You can read about them in Wikipedia (https://en.wikipedia.org/wiki/A/B_testing) or check this article with real examples: https://vwo.com/ab-testing/.

Here in Badoo, we have a lot of Split Tests running simultaneously, an example of which could be the appearance of the Own Profile tab.

One day, we decided that the profile looks a little bit outdated, and also we wanted to improve the engagement of some actions. So we run the Split Test with 3 groups:

  1. Old profile

As you see, there are 3 options, so this test is more like A/B/C (and that’s why I prefer the term Split Testing vs A/B Testing).

This is how different users saw their profile:

From the Product Manager console, it was 4 groups of users, distributed randomly and having equal size as follows:

You might ask, why do we have control and control_check (if control_check is the duplicate of control) here?

The answer is easy: any change impacts a lot of metrics, so you are never 100% sure if this change appears because of the split test or because of something else.

If you think that some metrics have changed because of the split test, you should double check that everything it is the same inside the control and control_check groups.

As you see… opinions might differ, but empirical evidence exists as a living proof; it is the responsibility of the product team to analyse the results and understand why one variant is better than another one.

Implementing Split Testing for your pet projects using Swift

Our goal:

  1. Make a client side framework (w/o external server)

P.S. Having a client side Split Testing framework brings both pros and cons. The main pros are that you shouldn’t need to have a server infrastructure or a server itself, but the cons are that if an experiment is a total mess, you can’t roll it back without uploading a new version to the App Store.

How we will do it:

  1. During execution, the group of the experiment will be assigned randomly with the same chances.

So let’s start:

This is what we are going to build.

We will have multiple split tests that will be represented by SplitTestProtocol, and every split test will have multiple groups (versions) that will be represented by SplitTestGroupProtocol. The Split Test will have the ability to hit (send to analytics) the current group, and so it will have AnalyticsProtocol as a dependency.

To fetch/generate/manage all split tests we will have SplitTestingService, because it will load the split test group from a memory that will depend on StorageProtocol, and also it will pass the AnalyticsProtocol dependency to the SplitTestProtocol.

Let’s write a code starting from AnalyticsProtocol and StorageProtocol dependencies:

The role of analytics is to log the event once, for example: log that user A is now inside group blue for button_color split test when he sees a screen with this button.

The role of storage is to save the value of the current user’s group after SplitTestingService has generated this group and read it every time it accesses it from code.

So let’s look at SplitTestGroupProtocol, and how it would represent groups.

It is RawRepresentable where RawValue is String, and so it will be possible to create the group from string or convert it back to string (useful for analytics and storage). Also SplitTestGroupProtocol contains a testGroups array, so it will be able to tell which groups it contains (and will be used to generate random groups).

Then let’s prototype the base for SplitTestProtocol:

SplitTestProtocol will contain:

  1. A GroupType that will implement SplitTestGroupProtocol protocol to represent the split test group type

⚠️ The hit split test method is very important to be sure that users are not just in a split test group, but also that the result of your split test has been seen. Marking the user as “saw_red_button_on_purcahse_screen” if he or she never visited the purchase screen would skew the results.

Now we are ready for SplitTestingService:

P.S. In this class we are using the Int.random function taken from https://stackoverflow.com/questions/25050309/swift-random-float-between-0-and-1, but in Swift 4.2 it would be by default.

This class contains 1 public method fetchSplitTest and 3 private methods saveGroup, getGroup,randomGroup.

The aim of randomGroup is to generate a random split test group for the current split test class, while getGroup and saveGroup will load split test instance with the specific group for the current user or save it to persistent storage for future usage.

The main and public function of this class is fetchSplitTest: it’s trying to return the current split test group from the persistent store, but if can’t — it will generate a new one and save it, before returning back.

Now we are ready, let’s create our first split test:

It looks big, but don’t worry: after you implement SplitTestProtocol by your class — all necessary properties will be requested to be implemented by compiler.

The important part here is the enum Group where you should put all your groups (in this example it’s red, blue and darkGray), and here we are defining the string to be sure how it will be reported to analytics.

Also we have extension ButtonColorSplitTest.Group to use all the sweet Swift syntax.

Now let’s create objects for AnalyticsProtocol and StorageProtocol:

We will use UserDefaults for StorageProtocol, because it’s easy to implement, but in your projects, you can use any other persistent storage.

For this example I will create a dummy analytics class, but in your project you can use real analytics. Personally I recommend Amplitude (https://amplitude.com).

Now we are ready to use our split test:

We are just creating its instance, fetching the split test and using it. Generics give us the ability to do buttonSplitTest.currentGroup.color.

During the first usage, you can see something like this Log once value: split_test-button_color for key: dark_gray and if you don’t delete the app from the device — the button will be the same on every launch.

Implementation process took some time, but after it every new SplitTest inside your project will take 1–2 minutes to create.

Here is an example of how this engine was used for a real app:

In analytics we segmented users according to different difficulty coefficients and how probable it is that the user will buy game currency.

So people who haven’t faced this coefficient (none) seemingly don’t play at all and don’t buy anything (which is logical), and this is why it’s important to hit split test when users really do face your test.

With no coefficient, just 2% of people bought currency. With the small coefficient 3% of people bought currency. And with the big difficulty coefficient, 4% bought currency.

So, if I enable the big coefficient for everyone, I can increase my profit :)

If you are interested in analysing results I recommend you check this tool: https://www.evanmiller.org/ab-testing/.

Thanks to my awesome team for helping me with this article (especially Igor, Kelly and Jairo).

The whole demo project is available via this link: https://github.com/azimin/SplitTestingArticle

Thank you for your attention!

Bumble Tech

This is the Bumble tech team blog focused on technology and…