How we implemented A/B(Experiment) service?

Photo by Fatos Bytyqi on Unsplash

Why A/B testing?

In business, sometimes we might have multiple solutions/features(flow in an app, colour, design) by our hypothesis and we may not be sure which one will work for the customers. So we decide to test them all out by releasing those features to a certain set of customers. We then analyse the data and make a calculated decision of which feature is serving best to our customers and decide to make it a permanent part of our app showing it to all users while deprecating other feature.

What is A/B testing?

A/B testing, also known as split testing, is a marketing experiment wherein you split your audience to test a number of variations of a campaign and determine which performs better. In other words, you can show version A of a content/feature/behaviour to one half of your audience, and version B to another.

Why not to use 3rd party services?

We did use 3rd party X for A/B.

With X, We observed the following problems:

• High latency because of server side tagging
• Feature complexity
• Tracking and Impact measurement of split sections.
• High cost

What our service offers?

This service (implemented as a micro-service) will enable clients to do A/B testing. We can do any number of experiments with multiple features(return values).

This is designed keeping in mind about scalability and further enhancements.

Common functionality includes:

1. Ability to control a feature go-live on a restricted basis (specific users)
2. Enabling any feature for random(%) users
3. Evaluating the feature on the basis of user profile/specific parameters with complex logic
4. Ability to control a front-end feature (UI element) on a restricted basis
5. Adding/Editing the features in any experiment
6. Capability to make segments mutually exclusive
7. Ability to find out which user fall under a particular segment
8. All other analyses that could stem from such experiment

Tech Stack:

Springboot, Java 8, Maven, Mysql, Groovy

Schema Design:

Here is a short description what above entities save. All entities extend BasEntity to get the common columns across all entities.

1. Experiment: All experiments will gets saved here with uniques name and optional description.
2. Feature: An experiment can have multiple features which will be saved in this table with unique name, optional description and experiment_id as foreign key to experiment table.
   Value is what that feature will return to the client in json format.
   Whitelisted_users will contain list of user_id as a comma separated list which will tell what users are assigned to this feature.
3. Logic: Since each experiment can have certain criteria to divide the user segment on some condition, this condition will be saved in condition column of logic table. (We will come to implementation later)
4. User_Feature_Mapping: Since we might want that once a user is assigned to certain feature, it will remain in same feature for future api calls. In other words, we might want to maintain consistency instead of evaluating logic(condition) every time.

Flow Design:

Now we will see what above 2 flows means:

The usePreCalculated flag is important here as it means whether we want to reuse the value contained in User_Feature_Mapping table or evaluate condition from logic table again.

1. If it is false, we will check in whitelisted_list of each feature of the experiment. If it exits then return value, else we will evaluate the logic and then accordingly return the feature value;
2. If it is true, we will first check in user_feature_mapping table if that user_id and experiment_id is present. If yes then we return the value. If not, then we will follow above step (step 1)

Note: Every time we evaluate the logic, we will save the result in user_feature_mapping table

How do we evaluate condition?

We are storing groovy code in condition column of logic table. Below is just one sample example:

import groovy.lang.GroovyShell;

public class GroovyDemo {
    public static void main(String[] args) {
        System.out.println("This represents some random code");

        String groovyScript = "println 'first line of Groovy output'\n" +
                "println 'second line of Groovy output'";

        GroovyShell groovyShell = new GroovyShell();

        // instead of passing a String you could pass a
        // URI, a File, a Reader, etc... See GroovyShell javadocs
        groovyShell.evaluate(groovyScript);

        System.out.println("This represents some more random code");
    }
}

We chose groovy shell for its sheer amount of flexibility. We can execute complex functions and even call external methods and apis with the code stored as a text. This is beneficial when our use case is complex such as we want to assign user segments on the basis of certain user properties or parameter which can only be fulfilled by 3rd party apis.

For example, if we want to divide the A/B as 90:10, we can do modulo 10 operation on user_id and assign result 0 to segment B and remaining to segment A. We can change the ratio or even logic any number of times without the need of redeploying.

List of APIs:

1. To create an experiment
2. To edit the experiment
3. To get the details of an experiment
4. To add a feature to an experiment
5. To edit a feature
6. To edit whitelisted users list for certain feature
7. To Evaluate an experiment and return response in json (Major Api)

Data Tracking:

In order to analyse our features’ performance, we need to store events in one place, be it DB or 3rd party like Mixpanel that will help us to create funnels and compare our features’ performance in any given timeframe to get insights and make a decision regarding which feature to be final.

Way ahead:

We can aim for introducing caching for optimisation of latency. Also using GroovyShell introduces a high waiting time in the queue if number of requests per unit time is high. This can be avoided by writing a general function in the code itself which will split the users according to the percentage given in the feature table while GroovyShell still can be kept for high flexibility it offers.