The limits of A/B testing
And why you can’t test your way to the creative solutions of tomorrow
Once upon a time
I previously worked at a company where A/B testing had been used very successfully to optimize the initial conversion funnel, and afterwards to optimize sales. It had in fact been so successful, that A/B testing had become the de facto way of validating any and everything.
Did customers prefer puppies over flowers? Where they into big buttons or small buttons? Was blue or orange better button colours? Should we have chicken or veal for lunch? (Okay, maybe not the last one)
A/B testing was used as the main driver for all product innovations. If you couldn’t create a small test to validate the need for a bigger solution, the bigger solution would not be created.
Initially that sounds like a fair proposal: If you can’t prove that the time spent on something will be well spent, then why should it be pursued at all? But, when you investigate further that argument is sometimes very flawed.
When is A/B testing great?
You love A/B testing. I love A/B testing. Everybody loves A/B testing! But, A/B testing is not a golden solution that cures all problems.
A/B testing is great when you are testing variations on similar solutions, and where you have one or more metrics that are clear indicators of success or failure.
A funnel is the most obvious example. You change a headline somewhere and can instantly measure whether more people got further into the funnel, whether more people converted at the end of the funnel and, for the ultimate result, whether the change generated a higher lifetime value.
Certainly a simple example, and A/B testing can be used on more complex solutions, but you get the picture.
Note: To get true learning out of A/B testing, and not just the information that variation A beat B, all variations should be based on hypotheses. Here a links to a couple of ways of working with hypotheses: Lean hypotheses, Writing Kick-Ass Hypotheses and The Problem With A/B Testing Success Stories
When is A/B testing the wrong tool for the job?
Anytime you face a complex problem, or anytime you want to prepare yourself for the future, then A/B testing is not the right tool.
Let me give some examples.
Any first version of something truly new will have basic flaws
You can’t A/B test your way to an original and innovative solution, because if it is truly innovative and original, then you are very likely to make initial basic mistakes that will make the test turn out negative, simply because the new path is unknown and you haven’t yet learned the common mistakes in this area. And even if the test is positive, you won’t know why.
Also, if you only change one part of the experience to fit the new flow (so the test is easier to create), you will never know if a failure is due to the new idea sucking, or it simply not being implemented far enough to provide the full usefulness you envisioned when initially thinking of it.
A/B testing can’t find taller mountains
You can use A/B testing to move higher and higher up on the mountain you are already on, but you can’t use it to spot a new mountain. If what your product really needs is a complete revamp, then all the A/B testing in the world can’t save it.
A/B testing two solutions with complex differences doesn’t provide any new learning
User tests are far more important than A/B tests at this stage. You want to find out exactly where the problems and advantages in each solution are, and get concrete ideas on where to improve, not just if A or B is better.
Non-funnel impacts may take a long time to measure
Let’s say you tweaked a part of your product and that user testing has shown that the new way of doing it works very well. If you do an A/B test and don’t spot any initial differences in conversions or up sells, was it worth the trouble? It might still very well be. If you really improved the product it should show of in higher retention and higher lifetime value, but these KPIs require long time periods and very big numbers to test reliably.
You have the high number of participants early in the funnel, maybe on your marketing landing page, but do you really have big enough numbers on a specific sub part of your application to measure anything within a useful timeframe? Unless you are Facebook, you probably don’t.
A/B testing can make you forget about the big picture
I know that all A/B testers don’t succumb to this problem, but I have seen it happen, so I am going to mention it: Focussing relentlessly on A/B tests sometimes makes people take their eyes of the horizon and only focus on the here and now. Sure, short term everything looks good, but if you don’t look up, don’t plan ahead, you are going to get hit by the bus of tomorrow sooner rather than later.
As I said earlier, I love A/B testing, but as with any tool you have to know its limits to use it well.
Note: I would love to hear what you think about this essay, so don’t hesitate to say hi on twitter: @madsbuchstage
