The order of experiments
Everyone is under time pressure, so finding the time to run every conceivable experiment is just not realistic. What we must do then is adopt a smart approach and find the cleverest order in which we should run the experiments within the experimental approach.
Overall philosophy
In general, the philosophy behind the experimental approach is to:
Validate your idea (and learn) in the fastest, cheapest way possible
The deeper we come into discovery, the more reliable the method we need. So initially, we might do internal user testing (and even do a quick survey), but to validate an idea, we need to bring in more reliable methods.
The steps in the diagram on the left should not be perceived as recommending any given order because I would often suggest running experiments as the first thing you do. While prototyping and interviews are strong methods in understanding users (which you will need, because experiments only give you very specific information about users, and do not give you incentives or thoughts), experiments give you more reliable data.
Different methods have different costs in different companies. Within IT, running online experiments are often pretty cheap, so running them early in a discovery process is an advantage.
The validation methods all have different advantages, but running an experiment is a must before you even think about building something new.
So how should we plan or structure our experimental approach?
Start with an easy proof-of-concept experiment
The general approach is to start with a simple proof-of-concept experiment (PoC). This is literally the easiest type of experiment you can do. I often have to remind both myself and our engineers that these types of experiments are allowed to be low fidelity; you don’t need a polished design. And it’s okay to have some friction in the user journey. The point is not to test a flashy feature or idea. The point is to see if users do what we expect them to (measuring behavior).
Example: Adding a rating box
An example of a proof-of-concept experiment can be seen above. For the online review website Trustpilot, we wanted to see if adding a box with unfilled stars would make users more likely to write a review. We could have tested different boxes, sizes, layouts and functionalities, but we only tested this simple one. It turned out to be a success. We increased conversion more than 10%. After this, we could test different variations on the same theme and increase conversion even more. If the PoC experiment had not worked, we would have stopped there without the extra effort.
Beginning with variations
When the advice “begin with PoC” is said, we often find ourselves in a situation where we run several variations in the Proof-of-Concept round. The reason is that the time it takes to set up the experiment itself is much longer than the little time it takes to add a few extra variations. We generally try to follow the formula Value / Work, and if a variation is almost no extra work, we add it to the PoC experiment.
Test your most risky assumptions
When you start running more experiments, the order of your experiments should be prioritized after your most risky assumptions. If you are testing an entirely new product or feature, there are multiple aspects of your idea that might fail. In this case, you should look at the entire concept, pin out all your assumptions, order them after how risky they are, and begin with testing the riskiest assumptions in different points of your product.
Read more about experiments in our Experiments publication.
