“A nightmare, a nightmare, and once again a nightmare” — but not a failure
You failed.
Doesn’t feel great to hear that, does it? So why should we use it to describe our experiments?
Failed is an emotionally loaded word. It also sounds very terminal. Implying a sense of finality (and shame) from which there’s no return. However, one of the joys of working in the internet economy is the never-ending opportunity to improve, iterate, tweak, or completely reinvent your product. All thanks to the stream of insights you get from “failed” features and experiments.
The other side of the coin, success, is not much better. It may be emotionally positive but it’s still a problematic word. It feels like you’ve finished, like you can leave that feature and move on to the next exciting one. Whereas in reality there are always ways to improve, always more you can learn. The internet is never finished.
Plenty of people have written about the need to embrace failure in order to advance. From John Maxwell’s Failing Forward and Seth Godin’s “If failure is not an option, then neither is success” to Facebook’s famous mantra of “Move fast and break things” (which they have now evolved into “Move fast with stable infra”). The greatest example of real-world commitment to this mind-set and approach for product development comes from Elon Musk, who is happy to write off millions of dollars in the name of learning.
My best personal example of an experiment that didn’t go the way we anticipated resulted in a 2-star Trustpilot review and a crushingly honest piece of customer feedback:
“Who came up with this nightmare? I could always see how much flights cost from all over the month. Now I see a nightmare. Most people chose the length of stay on the basis of ticket prices, not vice versa. I’ve never done otherwise. Now, to see the price of the month or at the turn of the two I have done dozens of searches. A nightmare, a nightmare, and once again a nightmare.”
And although we didn’t implement this feature the learnings we got from it were invaluable. Reams of data showing how people were using it, where their pain points were, and how they were instead bending other parts of the product to solve the job they needed to do. Coupled with insightful customer feedback and usability testing, allowing us to understand more about their expectations, mind-set and needs. From this we were able to improve that specific feature in different ways and apply the things we learned to other areas of our product.
So while I agree with the underlying philosophy of embracing failure I think referring to it as “failure” in the first place misses the point. At the very least it needlessly paints it in a negative light. If you insist on a “fail” sound-bite my favourite one is First Attempt In Learning. This gets to the heart of the matter and encapsulates the primary goal. The sole purpose of any experiment, new feature, or MVP test is to find out what works and what doesn’t. Whatever the result is, it arms you with the knowledge and insight to confidently take the next step. So I encourage you:
Stop failing and start learning.
Want to improve your A/B tests?
Check out my new site: ExperimentationHub.com/A-B-Sensei.html
Everything you need to know about A/B Testing and hypothesis driven experimentation! 👩🔬🎯 And all for free 😀
