Optimizing Experimentation Resources with Shared Control
Introduction
We see the benefits of shared resources in different parts of life. Before I talk about Shared Control, I would like to start with an analogy. Let us take an example of how Walmart “Pickup” service uses a shared resource.
Walmart Pickup: Users order online on Walmart.com and choose the option of pickup from store. One truck (the shared resource) ships items from a warehouse to the store for multiple orders placed on Walmart.com by multiple users. A user picks up his/her items at the store. So the truck is a shared resource used to deliver items of multiple users to the store.
In Experimentation we can use the same concept of sharing resources to increase efficiency of using traffic. Shared Control as its name suggests, refers to sharing one control as a shared resource across multiple experiments.
If you are new to A/B Testing and are not familiar with the terms like control please read along the next section.
Understanding The Basics
Before we dive into the specifics of Why, What & How of shared control, let us understand a few basic concepts.
What is a Control?
Control refers to the live version of your website/app. The control is the base against which all your website/app test results are compared.
What is a Variation?
The alternate experience of the website is called as the Variation.
What is a Treatment?
The type of change that is made to give you the altered experience of the website is known as the Treatment.
What is an Assignment?
The way A/B testing works is, some set of customers are assigned to the Live version (Control) of the website/app and some are assigned to the version with the treatment (Variation).
Assignment simply means that a certain customer gets randomly allocated either to see the control or variation treatment.
What is a Layer?
Expo, the A/B testing platform which we’ve developed and use in Walmart Labs, allows for customers to be assigned to multiple experiments. But we want to make sure that a customer doesn’t get assigned to experiments that modify the same content. This could end up spoiling the customers experience. To avoid such conflict expo organizes these experiments in such a way that, the ones modifying similar content are grouped together. And hence a user gets assigned to only one experiment in each group, so that his/her experience is not hampered. These groups are called as Layers.
Each Layer has a fixed capacity and hence only a limited set of experiments can exist within a layer based on how much percentage of space is occupied by each experiment.
How, Why & What ?
Let us continue with our Shared Control Story where we left off.
How did it all start ?
Early in the adoption of Walmart`s A/B Testing tool Expo, the number of experiments started to increase day by day, we started facing the problem of lack of space on a layer for running multiple experiments. The immediate solution to this problem was creating multiple layers of the same kind. But back then we did not have a UI for managing or creating these layers and the process of creating each layer was through an api call to the backend service of Expo. Requests started flowing in for more and more layers. Something had to be done about this situation.
Experimentation without Shared Control ?
Say a user creates multiple experiments and all of them affect Home page. Now Expo users select layers based on type of experiment. So, as the experiments are affecting Home page, the Expo user might usually select Home Page Layer for all those experiments. When Expo user sets up an experiment, he/she would usually create a Control (default experience) & maybe single or a couple of variations of that control. All together the Home Page Layer would end up having multiple controls and multiple variations.
Now when a customer visits the home page, they can get assigned to either of the Controls or variations of the experiments on the home page layer. Looking at the example below, consider a set of customers can get assigned to Control or Variation 1 of Experiment 1. Some may get assigned to Control or Variation 2 of Experiment2 and so on and so forth. Each of these controls & variation occupy a certain percentage of space on a layer. This was the only way assignment of traffic to experiments in a layer used to happen before shared control.
Why do we share the Control ?
While trying to find the solution for our lack of capacity per layer issue, we realized that there is lot of space that is being currently wasted.
We identified that set of customers that get assigned to Control 1 of Experiment 1 witness the same experience as set of customers that get assigned to Control 2 of Experiment 2. This brought up the issue of redundancy or not smartly utilizing the customer traffic. What this meant was the Control experience was exact same for all the experiments controls on the same layer but still each control was having its own different set of customer traffic. Hence occupying more space on the layer.
What did we gain from Shared Control ?
The most important thing we gained with Shared Control is traffic optimization. The space on a layer was occupied by traffic allocated to the controls of each experiment which are similar to each other. This traffic space was optimized with shared control.
Now the next question was how do we optimize the use of this traffic space on each layer?
We realized that all the experiments running on the same layer can share the same control and have the same set of customer traffic experiencing that control.
What this means is that instead of running 3 experiments with 3 Controls and 3 Variations (Assuming each experiment has just one variation), we can run 1 Control with those 3 Variations.
To understand this better, let us take an example with some numbers. The below images show controls of two different experiment which are identical to each other i.e. Experiment-1 Control and Experiment-2 Control. Each of which has a 40% traffic allocated. This adds up to 80% of traffic occupied on the layer.
Instead of having two different controls in both experiments. Each experiment can share the same control, hence saving the traffic occupied on the layer. So now instead of 40 + 40 = 80%, we have just the shared 40% of the traffic.
Shared Control in Expo
As mentioned earlier a Layer is a grouping of experiments done by Expo to avoid any kind of conflicts that can spoil customer experience while performing A/B Testing.
Shared Control allows more experiments to run concurrently by making more efficient use of available site traffic. It does that by allowing multiple experiments in a single layer to use the Shared Control Variation as their Control variation, instead of each experiment creating its own Control Variation & each with its own traffic allocation. Another thing to note about Shared Control Variation is that as it’s a control, hence it has no treatments. It acts as “default experience” for its layer.
Shared Control in Expo is configured at the Layer level. A percentage of Layer Traffic can be reserved to be Shared Control. Each experiment needs to be assigned a layer at configuring time. All experiments are run on layers in Expo.
Conclusion
When we started with Expo the number of experiments that ran were not many and hence initially, we never faced the issue of running out of layer capacity. But as the number of experiments running on a layer increased with more use of the tool and new users being added, the issue of layer capacity started to show up. So, if we intended to run more experiments on the same layer and it had no capacity, then we could not do anything. The only way was to wait for the current running experiments to end and then when the space was free, we could add more experiments to be run.
Just as Walmart uses trucks as a shared resource to ship items from warehouse to store for multiple people to save time and money. We used the same concept by sharing control and saving space to help run more experiments concurrently on a single layer. In the big picture Shared Control has been major contributor to optimizing layer space and also allowed more experiments to be run concurrently.
