How to build an experimental process

Those who start using Growth Hacking in their projects often make one mistake. They try to see what experiments were carried out in other companies and repeat them.

This simple and seemingly effective way doesn’t lead to positive results. What worked for some people doesn’t necessarily work for others.

For experiments to be useful, it is necessary to proceed not from solutions, but from the tasks that need to be solved. For the solved assignments to help you grow your business, you need a well-functioning process and a balanced approach to prioritizing experiments.

Extra tool: Notion — all-in-one workspace for note-taking, project management and task management.

Be free to send us any feedback by reply or reach using DM on Instagram

Do you want to receive insights using FB Messenger or Telegram or Email? 👈 Subscribe now.

Why do we even have to experiment?

Experiments are conducted to find the best solutions that will help to grow the main business metrics (North Star Metric). At the same time, without spending extra resources on the implementation of ineffective solutions.
There are two types of tasks that can be solved by growth experiments:

1. Optimize conversion at a specific point in the product;
2. Identify changes affecting the growth of metrics.

In both cases, it is necessary to track the further impact of experimental changes not only on the target metrics but also on other relevant indicators.
For example, if you want to increase conversion in registration, you need to track how the test group will further use the product and convert to pay-per-view.
Experiments should be based on robust hypotheses and a deep understanding of users.
Few people have the resources, time and data to test any exciting assumptions. You will have to choose the hypotheses that have the highest potential.

Documentation

Detailed documentation plays an essential role in the hypothesis selection process. It includes the results of previous studies, diagrams based on primary metrics, hypothesis formulations, and the purpose of new experiments.

Documentation of the process of experiment work helps to formulate more precise assumptions, assess the expected effect and potential success.

Documentation should be open to all stakeholders in the company. This will allow sharing knowledge within the team, as well as with other departments and managers responsible for business growth.

The process

Each experiment goes through a well-organized process that allows you to increase efficiency and achieve better results.

1. Formulating a hypothesis

First of all, a hypothesis should be formulated that will form the basis of the experiment.

2. Team Brainstorm

The hypothesis becomes the basis for an open discussion in the team, during which everyone can suggest their ideas regarding the details of the theory and how to test it.
The result of the brainstorming session is a list of ideas and possible experiments, as well as those responsible for further research and development of each operation.

3. Research

At this stage, the data already available are examined, interviews with users are conducted, and possible effects and nuances of implementation are discussed.
Several questions need to be answered:

- Will the work on the experiment affect other teams?
- Who should be informed about the test?
- Who should be consulted?
- What part of the product and users will be affected by the experiment?
- What will technical work need to be done?
- Do you need to go into the other teams’ repositories?

The primary purpose of the stage is to assess whether the experiment can have a sufficient impact on the business to cover the costs of its organization and implementation.

4. Justification

The person in charge justifies the team why this particular experiment should be started and what results it can bring.

5. Modeling

This step describes the critical components of the experiment: what it will be, which users will be in the test group, how much data will be required for statistical significance, and at what point the experiment can be stopped.

6. Prediction of results

The experiments are carried out not only to improve specific conversions and social metrics. The result of the test should be the growth of North Star Metric.
At this stage, specific results are forecasted, which the experiment should show at different levels of business.

7. Defining success criteria

An experiment should have a clearly defined criterion that determines its success. This can be either achievement of a particular result from the previous paragraph or an answer to an important question that will help the team to work more effectively in the future.

8. Expert judgment

When the experiment is fully formulated and described in detail, it goes through another filter.
Each member of the team should carefully study the information about the experiment and confirm that he or she agrees with all the fundamental theses.
It is essential that the discussion does not fall into the organizational details at this stage. The experiment is already well developed, and now only the key points and confidence in success are essential.

9. Planning and start-up

Sufficient understanding of the potential impact and the efforts required to launch the experiment has been obtained from studies, discussions, and assessments.
It is now necessary to plan the implementation and launch of the experiment.

10. Analysis

When the experiment is complete, the person in charge analyzes the collected data.
It is essential that all the necessary criteria defined during the simulation phase are meet and that the data are 100% transparent to other team members.
This will help to avoid p-hacking — distorting the results of the experiment in favor of confirming the original hypothesis.

11. Discussion of the results

At the next meeting, the team should discuss the latest results of the experiment, record the knowledge gained and plan the following steps.
During discussions, new hypotheses and ideas for experimentation are often born.
—
An iterative process of testing involving team discussions and several stages of filtering helps to formulate many hypotheses and to bring to the experiments only the best of them.