Creating a simple experiment
This article is part of my series about the experimental approach. In this article, we will give an example of how to run a simple experiment.
The idea and the pain
The experiment approach suggests that you should run experiments to assess the value of an idea. An idea often comes as a solution to a user’s “pain”/need. If we combine those two, we can call them the hypothesis: People have this pain, and the idea will solve the pain. (We could have spent time researching the user pain, but that is somewhat included in the experiment).
In Trustpilot, we had a hypothesis that users had trouble signing in (pain), and our solution was to add more social login buttons (idea). The “pain” was backed up by data.
Two things are important when running experiments:
- People should not know they are part of an experiment, as people often change behavior if they know what is being measured.
- You should measure behavior, not opinion (if possible).
So to experiment with this, we created three fake door login buttons, so people would believe that these social logins existed.
The ethical dilemma
There is an inherent problem in the experimental approach: We try to learn by making users think something exists, but we don’t want to mislead our users entirely.
Just like in psychology studies, if something is completely bogus, we should let our users know at the end of the experiment (if possible). In this experiment, we simply wrote a text box saying that it was an experiment after they clicked one of the fake door login options. Then we presented the actual possible login methods.
Assessment of experimental results
In this experiment, it turned out Google was quite a popular login method (We did run other experiment variations). When given a free choice, 18% of all users would choose Google to sign in. So this could bring our social logins to 46%.
We then had to assess if this is a successful result. Doing this is often not easy. We normally use this fraction:
Value / Cost
Specifically for IT, we look at it as value / work. Looking at our data, we could see that 1/3 of people who opt to sign in with email (which entails leaving our platform, going to their inbox and verifying their email) never return to our platform. So we thought that 18%/3 = 6% more logged in users was worth the work effort (approx two weeks).
Building on new learning
After having run an experiment, you are almost always smarter than before. We often combine these experiments with user interviews to understand the “why” of the behavior.
Read more about experiments in our Experiments publication.
