Fostering a Culture of Experimentation at Skyscanner

by Sophie Harpur

Skyscanner now has over 70 million monthly active users and operates in over 65 markets. At Skyscanner we use data and measurement to help our ever growing number of travellers achieve their goals. To facilitate our next stage of Skyscanner’s growth, we need to continuously build on our experimentation culture with our core engineering, growth, design and product principles in mind. Part of this is embracing testing as a part of our daily routine and incorporated into everything we do. Here are a few of the things we have implemented to make our experimentation culture a reality, at scale.

In-house Experimentation Platform

We have our in house experimentation platform, called Dr Jekyll. By investing in our own experimentation platform, we can get to the heart of our products, test assumptions and increase the frequency of testing for our users, impactful and valuable changes. Having our own experimentation platform also allows us to tailor the platform for the use of our internal users and specifically, the metrics they are interested in measuring which aligns with Skyscanner’s strategy.

Education through our UI
When you first join Skyscanner we don’t expect you to have a background or experience in running experiments. That’s why we aim to democratize experimentation through a user friendly UI which helps inherently explain the experimentation cycle. We use expertise from design, data science and engineering to build Dr Jekyll as an explorable learn-able tool to move people from their first steps of running an experiment to understanding the nature of A/B significance testing.

“I had basic knowledge on experimentation, but nothing like what we do at Skyscanner — Dr Jekyll has made this much easier for me to put things out to users and know the impact it can have. My squad is enabled and I am enabled to best serve the traveller.” — Harley Gribben — Product Manager

“At our scale even what seems like the smallest tweak or optimisation can have a large business impact. Our platform allows us to see these effects from bottom to top. This means Engineering can hold itself accountable, and move purposely but with safety, in exactly the same way as our other disciplines.” — Mike Moran — Engineering Manager

We don’t expect our Skyscanner users to understand the algorithms and calculations we use to show the cheapest travel deals. This also applies to our internal tooling, our internal users don’t need to worry about z-scores and alpha values that accompany experimentation calculations, that’s why try and automate as much of the process as possible so our internal users only need to input things that they care about. Ultimately we want our colleagues to get their experiments up and running as quickly as possible. Part of this includes our hypothesis kit.

We developed a ‘Hypothesis Kit’ that is easy to use, designed to guide experimenters with no or limited experience to on-board and start running trustworthy experiments quickly with the Skyscanner philosophy in mind;

Design like you’re right:

Based on [quantitative/qualitative insight].

We predict that [product change] will cause [impact].

Test like you’re wrong:

We will test this by assuming the prediction has no effect (the null hypothesis) and running an experiment for [X week(s)].

If we can measure a [Y%] statistically significant change in [metric(s)] then we reject the null hypothesis and conclude there is an effect.

This philosophy spans across product, growth and engineering. We have our philosophy front and center when a user logs in:

“Our culture of hypothesis-driven experimentation is so important and so successful because it allows anyone at Skyscanner to quickly test an idea. We no longer spend months creating a feature that may or may not work. Instead we test lots of small ideas, all based on insights and data, to see what potential they have. The more ideas we test, the more we learn about travellers, and the more informed our future tests are. And, of course, it helps us avoid the notorious HiPPO (Highest Paid Person’s Opinion)” — Rik Higham —Senior Product Manager (and creator of ExperimentationHub)

Changing our Mindset & Processes

Fail Forward
A big part of experimentation, is that ‘success’ or proven hypothesis rates are usually low — I mean, really low. But that is the nature of the beast and difficult to avoid, hence we encourage people to celebrate failure. “Fail forward” being something you will hear often around the office…we even got stickers with it! In experimentation a fail is as valid as a success, a failed experiment can provide insights that you may have not thought of before, and can make you one step closer to a win.

“While we all hope that we nail the user experience in the first implementation of a feature in reality this almost never happens. When this happens and the user behaviour isn’t quite what we’d wished it common to hear the experiment owner say that the experiment failed. I disagree, as long as you have learnt from an experiment and can build on that knowledge and either iterate or pivot with more confidence then the experiment did exactly what it was designed to do and helped you to understand cause and effect. An experiment is only a failure if you fail to learn, even if metrics appear to go up — if you don’t understand why the metrics changed you’ll never be able to recreate or build on your success.” — Dave Pier, Principal Product Manager

Peer Review
How do we ensure the quality of experiments while continuing to work autonomously and avoid those dreaded sign-off meetings? We created a culture and process of peer review. (Interested on how we developed a process of peer review for Growth, check out this blog-post). You can ask for a review of your hypothesis with a press of a button from Dr Jekyll, our experimentation platform, to our dedicated Slack channel.

Peer review also allows for visibility of experiments across squads and tribes, and allows our colleagues to bring back learnings to their respective squads. This an effective process for fast moving, highly autonomous teams.

“Peer review serves as a checks and balances system that ensures a highly effective experimentation process. It cultivates a culture of continuous knowledge sharing by allowing peers to challenge and understand experimentation decisions.” — Marissa Hills — Growth Manager

External Inspiration
Nothing works better for inspiration than a fresh perspective from outside of Skyscanner. We have been lucky in the past to have some amazing speakers to come talk to Skyscanner about experimentation within their organisation including; Ben Dressler from Spotify, Lukas Vermeer from Booking.com and Jane Murison from the BBC. (Psst are you a pro at all things experimentation? Would you be willing to speak at a session? We would love to hear from you!). We also encourage our colleagues to attend conferences that have a focus on experimentation where they can also obtain new perspectives and learn new industry standards to bring back to the organisation.

Education at Skyscanner University

Experimentation 101
Through our Skyscanner University programme, employees have the flexibility to choose what courses and personal development they’d like, courses range from coffee making to Mandarin but Experimentation 101 is also a popular choice. We ensure that the education we provide is interactive and accessible to people from all functions and disciplines. We also ensure this course is relative and can be applied on a daily basis, not just purely theory.

Additional Support We Provide
Some of the additional support we provide for experimentation are dedicated Slack channels for experimentation discussions, we also have drop-in sessions if people prefer to ask their experimentation questions face to face. We also show case different “Experiments of the Week”, this circulation is one of the ways we gain visibility from across the different offices around the world.

The experimentation culture at Skyscanner has benefited disciplines across the business, design, growth, engineering and product. We are excited to continue developing our world-class experimentation standards and create better and better user journeys for our travellers.

Grow with us

Take a look at our current job roles available across our 10 global offices.

About the Author

Hi I am Sophie and I am part of the Experimentation Services Squad at Skyscanner. We enable our country squads to run lean and agile activities from channel tests to A/B product experiments. Skyscanner is a fantastic place to work and challenges me regularly in new and exciting ways.