Fostering a Culture of Experimentation at Skyscanner

by Sophie Harpur

Skyscanner now has over 70 million monthly active users and operates in over 65 markets. At Skyscanner we use data and measurement to help our ever growing number of travellers achieve their goals. To facilitate our next stage of Skyscanner’s growth, we need to continuously build on our experimentation culture with our core engineering, growth, design and product principles in mind. Part of this is embracing testing as a part of our daily routine and incorporated into everything we do. Here are a few of the things we have implemented to make our experimentation culture a reality, at scale.

In-house Experimentation Platform

Education through our UI
When you first join Skyscanner we don’t expect you to have a background or experience in running experiments. That’s why we aim to democratize experimentation through a user friendly UI which helps inherently explain the experimentation cycle. We use expertise from design, data science and engineering to build Dr Jekyll as an explorable learn-able tool to move people from their first steps of running an experiment to understanding the nature of A/B significance testing.

“I had basic knowledge on experimentation, but nothing like what we do at Skyscanner — Dr Jekyll has made this much easier for me to put things out to users and know the impact it can have. My squad is enabled and I am enabled to best serve the traveller.” — Harley Gribben — Product Manager

“At our scale even what seems like the smallest tweak or optimisation can have a large business impact. Our platform allows us to see these effects from bottom to top. This means Engineering can hold itself accountable, and move purposely but with safety, in exactly the same way as our other disciplines.” — Mike Moran — Engineering Manager

We don’t expect our Skyscanner users to understand the algorithms and calculations we use to show the cheapest travel deals. This also applies to our internal tooling, our internal users don’t need to worry about z-scores and alpha values that accompany experimentation calculations, that’s why try and automate as much of the process as possible so our internal users only need to input things that they care about. Ultimately we want our colleagues to get their experiments up and running as quickly as possible. Part of this includes our hypothesis kit.

We developed a ‘Hypothesis Kit’ that is easy to use, designed to guide experimenters with no or limited experience to on-board and start running trustworthy experiments quickly with the Skyscanner philosophy in mind;

Design like you’re right:

Based on [quantitative/qualitative insight].

We predict that [product change] will cause [impact].

Test like you’re wrong:

We will test this by assuming the prediction has no effect (the null hypothesis) and running an experiment for [X week(s)].

If we can measure a [Y%] statistically significant change in [metric(s)] then we reject the null hypothesis and conclude there is an effect.

This philosophy spans across product, growth and engineering. We have our philosophy front and center when a user logs in:

“Our culture of hypothesis-driven experimentation is so important and so successful because it allows anyone at Skyscanner to quickly test an idea. We no longer spend months creating a feature that may or may not work. Instead we test lots of small ideas, all based on insights and data, to see what potential they have. The more ideas we test, the more we learn about travellers, and the more informed our future tests are. And, of course, it helps us avoid the notorious HiPPO (Highest Paid Person’s Opinion)” — Rik Higham —Senior Product Manager (and creator of ExperimentationHub)

Changing our Mindset & Processes

“While we all hope that we nail the user experience in the first implementation of a feature in reality this almost never happens. When this happens and the user behaviour isn’t quite what we’d wished it common to hear the experiment owner say that the experiment failed. I disagree, as long as you have learnt from an experiment and can build on that knowledge and either iterate or pivot with more confidence then the experiment did exactly what it was designed to do and helped you to understand cause and effect. An experiment is only a failure if you fail to learn, even if metrics appear to go up — if you don’t understand why the metrics changed you’ll never be able to recreate or build on your success.” — Dave Pier, Principal Product Manager

Peer Review
How do we ensure the quality of experiments while continuing to work autonomously and avoid those dreaded sign-off meetings? We created a culture and process of peer review. (Interested on how we developed a process of peer review for Growth, check out this blog-post). You can ask for a review of your hypothesis with a press of a button from Dr Jekyll, our experimentation platform, to our dedicated Slack channel.

Peer review also allows for visibility of experiments across squads and tribes, and allows our colleagues to bring back learnings to their respective squads. This an effective process for fast moving, highly autonomous teams.

“Peer review serves as a checks and balances system that ensures a highly effective experimentation process. It cultivates a culture of continuous knowledge sharing by allowing peers to challenge and understand experimentation decisions.” — Marissa Hills — Growth Manager

External Inspiration
Nothing works better for inspiration than a fresh perspective from outside of Skyscanner. We have been lucky in the past to have some amazing speakers to come talk to Skyscanner about experimentation within their organisation including; Ben Dressler from Spotify, Lukas Vermeer from Booking.com and Jane Murison from the BBC. (Psst are you a pro at all things experimentation? Would you be willing to speak at a session? We would love to hear from you!). We also encourage our colleagues to attend conferences that have a focus on experimentation where they can also obtain new perspectives and learn new industry standards to bring back to the organisation.

Education at Skyscanner University

Additional Support We Provide
Some of the additional support we provide for experimentation are dedicated Slack channels for experimentation discussions, we also have drop-in sessions if people prefer to ask their experimentation questions face to face. We also show case different “Experiments of the Week”, this circulation is one of the ways we gain visibility from across the different offices around the world.

The experimentation culture at Skyscanner has benefited disciplines across the business, design, growth, engineering and product. We are excited to continue developing our world-class experimentation standards and create better and better user journeys for our travellers.

Grow with us

About the Author

We are the engineers at Skyscanner, the company changing how the world travels. Visit skyscanner.net to see how we walk the talk!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store