Demystifying Experimentation for Data-Driven Decisions
Welcome to a short article where I will be demystifying business experimentation for data-driven decisions. In this article, I will be introducing a few key terms regarding experimentation, its steps for the most common test (AB testing), and a case study.
At the end of this article, you will be able to identify what experimentation is and why it is important, understand the practical steps of the most common experimentation. You will also be able to define and analyze the variations between different types of experimentations. If you are ready, let’s get started.
Before diving into the details, just now I need you to pause reading and think of an answer to the following questions:
How do you know a company is innovative? Is it infrastructure? Is it people? Is it ideas?
I am sure your answers are most likely part of the innovation. But research shows that a simple and easy measure of innovation is this: “how common are experiments” in a company.
Because I am working in the travel industry as a data scientist, one of the most relevant examples for testing and experimentation I often give my colleagues is Booking.com.
Per year, Booking.com runs more than one thousand rigorous tests simultaneously and by an estimate more than 25 thousand tests according to Stefan Thomke. Booking.com runs such a big number of experiments not because they have enough infrastructure, data pipelines, and tens of hundreds of data scientists but because they have a strong culture of experimentation and empower every employee to run such tests independently.
What is Experimentation?
Experimentation in a simple term is to test ideas scientifically. There are a couple of steps you need to know and take before designing an experiment:
- Goal: you will need to establish the goal of your experiment. For instance, if you want to increase profit, define by how much and by when you’re looking to grow the net income of your company.
- Hypothesis: Afterward, you will have to define a hypothesis in which you consider whether one action at your place of business will lead to the goal or result that you are looking to achieve.
- Types of experiments: Then you’ll need to decide on the type of experiment you want to utilize to test your hypothesis. The most common experimentation is AB testing. A/B testing (also known as split testing) is a process of showing two variants of the same web page to different segments of website visitors at the same time and comparing which variant drives more conversions.
- Key Metrics for measurement: Then you’ll need to figure out how to define the key performance indicators (KPIs) in order to accurately measure the success of your experiment.
After you successfully design your experimentation, you will figure out if the treatment is the winner.
What are AB Testing and Multivariate Testing?
The simple A/B tests, in which two versions of a website can be compared, can be set up within a matter of days, and usually, last at least a week. There are third-party tools designed to allow every employee to test ideas.
For more rigorous tests, your data scientists can help you design the experiments and analyze the results.
Multivariate testing is another common type of experimentation: While it is similar to AB testing, this method allows you to test more than two changes at a time. You need to democratize experimentation in your company, team, or project.
Grow an Experimentation Culture across the company
Commitment is key when it comes to benefit from the transformative power of experimentation. Over time, experimental culture will generate huge benefits in the long run for your company.
