How to run an experiment

In my previous article I argued for more room to learn within product teams. This article gives direction to the practical implementation.

Navigate on measurable outcomes

As a designer, developer and product manager, we all want the same; make products that people like and want to use. During our work we make hundreds of assumptions every day. Assumptions regarding the user and his or her goals and benefits. We consult sources such as analytics and customer support. Finally, there is the business with all kinds of new ideas and that’s how we feed the monster: the product backlog.

User stories are written, prioritized and implemented based on all this input. Sprint after sprint after sprint. User stories are extremely suitable as a source of discussion and for estimating work. What we do not record in stories is the intended measurable improvement that we expect. This makes it extremely difficult to determine afterwards whether we have been successful in achieving our user needs and business goals.

Test cards, learning by experimenting

For the use of experiments in teams I use the test and learning cards from Alex Osterwalder. The test card helps you to formulate an assumption and forces you to make the following things explicit;

• Hypothesis: what must be true to validate the assumption?
• Solution: how are you going to test whether your assumption is true or false?
• Measure: what will you measure to validate the assumption?
• Criteria: when are we successful, what is the threshold?

Let’s start from the beginning. Before we can think about measurable outcomes, we have to take a good look at our assumptions. To test an assumption there are a number of preconditions that you must take into account.

Preconditions for testing

The hypothesis

Drawing up good assumptions (or hypotheses) is difficult for teams. Most people are more interested in solutions than problems. This leads to too many features and products with too little commitment.

Focus is important because you want to be able to properly validate what you are doing.

A good hypothesis / assumption contains the following building blocks.

We believe that [target audience] will take [action] for [reason]

Every key element in this sentence is a variable for your experiment and a possible reason for failure. It is important that you start the sentence with “we believe that” because you are immediately in the right mindset.

Example:
We believe that people that like cycling will book a cycling package sooner if they discover that many other people have booked the same package.

The test

Now it’s time to think about the solution that we have in mind. What does it look like and above all, how are we going to test it? Try starting your solution with:

To prove that we will…

Example:
To prove that, we make a widget with the most trending packages.

Measuring

What metric will we influence with our test? It is important that you clearly understand what the current baseline is. If you do not have reliable data, first make sure it you collect it before you start the experiment. Never start an experiment without reliable data.

Example:
The average number of viewed packages up to a booking.

Criteria

What is an acceptable improvement to be able to conclude that the experiment was successful? It’s not easy to paste a realistic number on a number that provides sufficient evidence for our assumption.

Example:
Reduce the number of viewed packages to a booking by 20%.

In practice you will see that a single experiment is not always sufficient to validate an assumption as true or false. This is not a problem. Every experiment you conduct learns you more about the possible next steps than simply finishing up user stories.

By working on outcomes instead of solutions you ensure that the team is focused on progress.

=======

Want to try your own experiment? Download the experiment card on our (Dutch) website.