How Booking.com A/B Tests like Nobody’s Business

Jenny Zhang
Taplytics Types
Published in
5 min readMay 17, 2017

Every year, Google holds a conference focused on sales conversions at their EMEA headquarters in Dublin. The theme for 2017 was “converting mobile consumers,” and two presentations stood out for us in particular: Stuart Frisby from Booking.com & Vanja Mlaco from Transavia. Both companies play in the travel/tourism space and stress the importance of testing culture within their organizations.

“If it can be a test, test it. If we can’t test it, we probably don’t do it.” — Stuart Frisby

First, we’ll be taking a look at how Booking.com as a company is fundamentally built around A/B testing. They are a travel fare aggregator site that connects consumers with overnight accommodation — over one million rooms are reserved daily. We couldn’t agree more with how Booking.com is approaching optimization. At the core of the user experience is personalization that ultimately leads to retention, and they are using testing as a means to achieve this.

The Importance of Testing

Conversion rate optimization (CRO) is centred around increasing the percentage of mobile/website visitors into customers. A/B testing is meant to help create the best version of a product that’s specifically tailored towards the consumers’ needs and wants. What appeals to your end consumer, and how exactly do you optimize your app for the largest group of people?

“Go in a direction where your customers are telling you that you’re adding value to their lives.”

Booking.com has been testing for 8 years, and initially built their own tools due to a lack of options on the market at the time. After dedicating funding and resources towards developing their testing processes, they created a strong in-house solution.

According to research by Evercore Group L.L.C., their “testing drives conversions across the whole platform at 2–3 times the industry average.” They are constantly running experiments, whether it be for performance improvements or new product features.

The Process

Booking.com’s Testing Process

Booking.com’s process is dependent on the fact that they have the resources to A/B test in house. Employees are encouraged to come up with hypotheses based on existing data that need to be proved or disproved. There is an emphasis on having a hypothesis as opposed to only an idea, because they’re meant to be “more grounded in the reality of the business.” As Stuart put it,

“People with ideas are artists, people with hypotheses are designers.” — Tweet This

After the hypothesis, the test is broken down into atomic experiments. Testing small ensures less risk, and gives more reliable results since there are fewer interfering factors. In the testing phase, there are 1000 experiments running at any time. To put it into perspective, there are more versions of the Booking.com website live than there are humans that have ever lived. It’s pretty hard to wrap your head around it!

Example of how an A/B test could be broken down into several iterations

“A/B testing is predicated on the idea that you have to trust your tools…You either need to trust the tools that you have, get better tools, or build your own tools.”

A lot of companies often wonder whether it’s better to build a testing platform in-house, or outsource to an external solution such as Taplytics. Booking.com is unique due to their business model — they don’t need to worry about supply chain management or maintaining their own inventory, and thus decided to make the necessary investment towards internal testing. However, they were also determined to take on this substantial project of developing in-house, and poured in the necessary time and effort. Booking.com made this decision 8 years ago, and it has definitely paid off.

When considering A/B testing, it’s important to evaluate whether your business has enough resources to build out all the software, or if the smarter choice is to look elsewhere for faster results.

Culture of Experimentation

Creating an A/B testing mindset is impossible without the support and participation of the whole organizational team. Employee input is highly valued, and there is as much transparency as possible when it comes to data access.

Booking.com believes in “guidelines, not rules,” and offers employees complete freedom to decide how and what to test. They throw HIPPOs (highest paid person’s opinion) out the window in favour of increased democracy within company decision making. There are 75 front-end product teams each with their own multi-disciplinary components (eg. one designer per team), which allows them to zone in on specific problems. Employees also change teams every 10–12 months in order to see the company through a more holistic view and develop a flexible mindset.

A company that recognizes the power of A/B testing is well-positioned for the future. This is applicable to all industries outside of travel, whether it be sports entertainment or retail e-commerce. Here at Taplytics, we’ve seen some amazing results from our clients due to their willingness to experiment and back up hypotheses with concrete data. Booking.com’s approach to optimization is something we hope to see more companies adopting as mobile interactions grow to define the consumer experience.

As Stuart Frisby said:

“Throw out your roadmap. Trust your people. Trust your tools. Trust good science.”

Check out the presentation below:

Taplytics is a fully integrated mobile A/B testing, push notification, and analytics solution providing the tools you need to optimize your mobile app.

--

--