What and When Should I Mobile A/B Test?
Last week we ran an event with Nima Gardideh of Frank & Oak hosted by Pivotal Labs. It was a great event where Nima talked about Frank & Oak’s culture of testing. While all of his insights were great, one thing in particular stuck out for me. That Frank & Oak has made the conscious decision that any new feature must be introduced as an A/B test. I always knew Nima and Frank & Oak had a solid culture of testing, but I never knew they had such dedication to it. This is in stark contrast to the preconceived notion that to start mobile A/B testing you need a minimum level of technology or a stabilized growth rate, or that only certain things can and should be A/B tested, like buttons, text and images.
I thought the talk was great, because it answered the common question I get of what should be A/B tested and when A/B testing should start. Nima’s answers were: everything and immediately.
The Odd Nature of Feature Releases
It’s understandable why people may be hesitant to A/B test early and on all parts of an app. When you just ship a new feature without a test you know what the result will be, the result is that your app now has a new feature. Whether good or bad for the overall goals of your company, as a product manager you can point to that new feature and say that was completed and progress was made.
When it comes to mobile A/B testing, the results are less defined from the outset. As you embark on your testing strategy you can’t know what the result will be, or even if the test itself will provide a clear winner. So I think there is a fear that a test will come back inconclusive, and the owner of that test won’t know how to answer the question of whether there was value in the test.
Rarely, though, have I seen a test that is truly inconclusive. The data almost always gives some insight, whether positive or negative, into the hypothesis in question. And knowing comparatively, whether one implementation is better or worse than another is entirely positive. Because when you release without an A/B test it is entirely possible that you are releasing a bad implementation, but you have no real way of knowing.
Everything we do is a Hypothesis
That’s why I particularly like Nima’s position of A/B testing everything, because really, everything that we do is just a hypothesis. We can never know ahead of time whether or not something will work the way we expect it to. So maybe start thinking the way Frank & Oak does and look at every new feature and every change in your app as a place for an A/B test. When you have an internal discussion about features or implementations, instead of making an either or decision, put the debate out to your users through a test and let them decide.
Feel free to put your thoughts on the topic in the comments below. What are your thoughts on when and what to A/B test?
