On A/B Testing: Preview
Ensuring the expected behavior of A/B experiments
A/B testing allows you to run experiments that change the behavior of a user’s experience. In a typical experiment you will have one group of users who will see the current, unmodified behavior (control) and another group of users see the modified behavior (variation). Whenever you’re altering behavior that has potential to effect your users experience it is a (really) good idea to be sure the expected changed behavior is the actual behavior the user sees. The tool used by A/B testing systems for giving this assurance is called Preview.
Preview, you mean like a movie preview?
Just like a movie preview, previewing an experiment shows you changed behavior before the experiment is released to the end user. Preview basically puts you into the experience that the user will see. This provides a visual assurance that things are working (or not). In some cases you may want to preview into the control to ensure the modified behavior is not showing.
Complex environments, such as a world class global e-Commerce website, will have many integrated parts and systems. Preview is the end-to-end validation that all the parts are working together as expected.
At Walmart we have had many experiments that were days and even hours out from going live only to be pulled back because the preview wasn’t showing the expected changed behavior.
Why is Preview important?
Preview gives experiment owners the power to verify your behavior in production before starting the experiment.
Preview allows experiment managers to find issues in the experiment setup, or issues in the development implementation or even failures in end-to-end integration BEFORE putting this behavior out for users.
Preview vs Multi Preview vs Opt-In
At Walmart we have a couple interchangeable terms when referring to experiment preview. For one we have a dedicated team that manages the experimentation process at Walmart. The experimentation team likes to use the term “Opt-In” to an experiment. Typically one thinks of opt-in as that little checkbox on a signup form that says you are allowing some business to spam you emails of sales or newsletters. In our context the opt-in is just a way of forcing yourself or whoever is setting up the experiment into a variation to allow them to see the modified experience.
Walmart’s A/B experimentation platform is named Expo. Expo has, in addition to preview, a multi-preview option. Multi-preview allows an experiment manager to select multiple variations setup on different experiments (which are mutually exclusive, meaning not overlapping) and preview into all at once. For example, there might be an experiment running on the home page of Walmart.com and a different experiment running on a category page and yet another experiment running on the shopping cart checkout. Multi-preview allows the user to opt-in to all these variations at one time so they can verify the user experience across the verticals.
How does Expo implement preview?
Without going deep into the magic of Expo, through a UI tool we allow users to manage experiments as well as preview or opt-in to an experiments control or variation. When the user clicks to preview a variation, Expo sends them over to Walmart.com with a URL parameter flag which tells our experiment assignment engine that the variation should be shown.
From the Expo UI the way to preview a variation looks like this.
Clicking the preview icon will send you over to Walmart.com showing the changed behavior. The image below is a mock experiment where the variation has an altered delivery message with a personalized name.
The Preview Assurance
Running experiments in a production environment should be ensured just like any other change you would push out to production. Just like you wouldn’t push out a development change without passing unit, functional, and performance tests. Preview is the way to make sure the changed behavior will not present a negative or unexpected experience to the user. I hope this helped show the importance of previewing your A/B test experiments. Now go test some previews!
