Toxic A/B tests
Why fast experiments make you slow
By Hilary Roberts
In 2015, we started running our first controlled A/B tests at Skyscanner, transforming the work of the Product team and the speed and confidence with which we could test new ideas. Since then, we’ve grown our experimentation to hundreds of A/B tests per month.
Each of these tests has been an opportunity for us to iterate through the build-measure-learn loop of Eric Ries’ Lean Startup, learning more about who our end-users are, what problems are most important to them, and whether the software we’ve built is a good solution.
The more experiments we run, the more we’ve started to understand that solving big, valuable problems is about more than just being data-driven, and that sometimes, the best way to eliminate waste from your development cycle is to avoid running any experiment at all.
I originally gave this presentation at the Edinburgh Turing Festival, and then updated it to include even more practical guidance for product practitioners at Canvas Conf.
You can watch the full Science and Sensibility talk below or on their website. You can also follow along with the slide deck.
Want to work with us?
We’re on the look out for Associate Product Managers and a Principal Design Manager to be based in Barcelona, Budapest or London.
We’re also looking for a Chief Information Security Officer and Senior Technical Managers. With a range of perks like Home country working, flexi-working, a learning environment and an authentically great company culture Skyscanner is a fantastic company to work for. Come and help us solve the complex problems that live in Travel.
About the author
Hilary cut her teeth in product working with dozens of startups from the University of Edinburgh to test their value propositions and find their first customers.
In 2013 she moved to Skyscanner, one of the world’s largest travel search sites. She is now product manager for the Flights Group, the company’s largest business vertical, with more than 50 million users per month globally.
Originally published at codevoyagers.com on February 24, 2017.
