How Booking.com A/B Tests Ten Novenonagintillion Versions of its Site
“If it can be a test, test it. If we can’t test it, we probably don’t do it.” — Stuart Frisby of Booking.com
Booking.com is a company that was built on A/B testing.
They are a travel fare aggregator site that connects consumers with overnight accommodation through which over one million rooms are reserved daily. At the core of that user experience is personalization, which is what ultimately brings people back to the site and gets them to make a purchase — and they find the perfect type of personalization to use through A/B testing.
Because Booking.com is an aggregator, they don’t need to worry about supply chain management or maintaining their own inventory. Instead, they obsessively focus on A/B testing — today, they may be the most sophisticated A/B testing company in the world, with 1,000 experiments running at any time.
One experiment forks a page into two variations, a second experiment forks those pages into four variations, a third experiment creates eight pages, and so on. With 1,000 experiments running at any one time, that makes as many as 2¹,000 — or ten novenonagintillion — different versions of the site.
To put that into perspective, there are more versions of the Booking.com website live right now than there are humans that have ever lived.
For Booking.com, this kind of testing is an everyday reality because it’s been built into the culture of the company from the very beginning.
Booking.com’s Culture of Experimentation
Creating an A/B testing mindset is impossible without the support and participation of the whole organizational team. Employee input is highly valued, and there is as much transparency as possible when it comes to data access.
Booking.com believes in “guidelines, not rules,” and offers employees complete freedom to decide how and what to test. That means:
- HIPPOs (highest paid person’s opinion) get thrown out the window in favor of increased democracy within company decision making.
- The 75 front-end product teams each have their own multi-disciplinary components (e.g. one designer per team), which allows them to zone in on specific problems.
- Teams change every 10–12 months in order to see the company through a more holistic view and develop a flexible mindset.
A company with A/B testing ingrained in its DNA will continue to learn, evolve and improve the product. This is applicable to all industries outside of travel, whether it be sports entertainment or retail e-commerce.
At Taplytics, we’ve seen some amazing growth from companies that are willing to experiment and back up their hypotheses with concrete data. It’s certainly paid off for Booking.com. According to research by Evercore Group L.L.C., Booking.com’s “testing drives conversions across the whole platform at 2–3 times the industry average.” That means massive increases to their revenue and bottom line.
Booking.com’s approach to optimization is something we hope to see more companies adopting as mobile interactions grow to define the consumer experience.
According to Stuart Frisby of Booking.com, to succeed with A/B testing, “Throw out your roadmap. Trust your people. Trust your tools. Trust good science.”
The A/B Testing Process at Booking.com
Conversion rate optimization (CRO) is centered around increasing the percentage of mobile/website visitors into customers. A/B testing helps discern the best version of a product that’s specifically tailored towards the consumers’ needs and wants.
What appeals to your end consumer, and how exactly do you optimize your app for the largest group of people? — that’s what you figure out through your A/B testing process.
Booking.com’s process is dependent on their company culture around A/B testing. Employees come up with hypotheses based on existing data that need to be proved or disproved. Those hypotheses feed into the A/B testing pipeline.
There is an emphasis on having a hypothesis as opposed to only an idea because they’re meant to be “more grounded in the reality of the business.” As Stuart put it,“People with ideas are artists, people with hypotheses are designers.”
After the hypothesis, the test is broken down into atomic experiments. Testing small ensures less risk, and gives more reliable results since there are fewer interfering factors.
Those tests are then run on Booking.com’s in-house A/B testing tool suite. Because they’ve been testing for 8 years, they initially built their own tools due to a lack of options on the market at the time. After dedicating funding and resources towards developing their testing processes, they’ve created a strong in-house solution.“
Many companies often wonder whether it’s better to build a testing platform in-house, or outsource to an external solution such as Taplytics. It’s not to be underestimated how much work is involved building an in-house solution.
As Stuart puts it, “A/B testing is predicated on the idea that you have to trust your tools…You either need to trust the tools that you have, get better tools, or build your own tools.” To get to the point where you can trust your tools, you need to pour in the equivalent of Booking.com’s 8+ years of in-house analytics development work.
Use A/B Testing to Follow Your Customers
To succeed with A/B testing, make it part of your company culture. Once you do that, have your culture feed and sustain a well-defined process.
What’s key is the fundamental understanding that A/B testing builds value. In Stuart’s words, A/B testing means “go[ing] in a direction where your customers are telling you that you’re adding value to their lives.” By following your customers via A/B testing, you can’t go wrong.
