Using the Optimizely API to avoid flashes of unstyled content in A/B tests
We use Optimizely on the Mozilla Developer Network to build and analyze split experiments. We find that the tool helps us move forward with confidence, understanding how changes affect user behavior. We care about accuracy and user experience, so we were concerned when we discovered a problem in a recent experiment that could have affected both.
Thankfully, there’s a simple way to avoid these flashes: modify styles with server-side code instead.
Create an experiment as usual and add some named variations. Don’t use the Optimizely editor to change any styles. Instead, use server-side code to show different styles to different users. We use percent-based feature flags to accomplish this, but you could alternatively show different styles randomly. You don’t even need to group users evenly; Optimizely handles unequal groups just fine.
How can you compare the behaviors of the different groups? The Optimizely API provides a method for this. Use bucketVisitor to manually group visitors into appropriate variations for later review.
The end result will look something like this. This example uses Jinja, but any server-side template language will do. Remember that this just changes styles and groups visitors. You’ll still need to build and run a corresponding Optimizely experiment with named variations. Be sure to update the second and third arguments of
push to use your real experiment and variation IDs. These are available in the experiment’s Diagnostic Report.
Need a real-world example? Take a look at this recent change to MDN that uses the same approach.
If you’re having trouble with flashes of unstyled content in Optimizely experiments, try this out. With just a little code, you can run completely invisible experiments without sacrificing any of the great analysis features Optimizely provides.
Originally published at blog.openjck.com on April 7, 2015.