The experimental approach

Since the Age of Enlightenment people have been conducting experiments to learn and gather information. Inside organizations, we often listen to senior managers as the main source of information. This is risky. Instead, the experimental approach gathers information through market experiments to answer one question: Is this a good idea?

This is the introduction to a series of stories about experiments and the experimental approach. The principles of this approach are:

• Run experiments to learn if something is a good idea.
  Further reading: Why traditional idea evaluation fails
• Start with simple experiments, then expand.
  Further reading: The order of experiments
• Run a kill experiment to know if users will buy your idea.
  Further reading: The kill experiment

Why experiments?

In all companies I have ever been in (from startups to enterprises), there have been plenty of ideas. The problem is to figure out what the best idea is, and use your time on that. Since the Age of Enlightenment people have been conducting experiments to learn and gather information. This is a robust methodology based on a scientific falsification approach. You have an idea (hypotheses) and you gather information from actual users (the market). For some reason, many of us stopped gathering information ideas when we started to work for corporations. We expected knowledge to be inside the organization, and the people with pointy hats in the top of the hierarchy should possess the most knowledge to assess if it’s a good idea. Experiences from my entire career find this to be untrue. The market is difficult to predict and understand.

The goal of the experimental approach is to:

• Use experiments to understand if something is a good idea (and learn from this understanding).

Is it a good idea to build a wall in our apartment in front of the staircase? Let’s do an experiment to learn, if it is.

Take a look at the example above: We were considering building a wall in our apartment in front of our staircase. We could have had endless discussions about light, heat, and airflow, but instead, we used the experimental approach to figure out if it was a good idea: We bought some cheap rolls of cover blankets and within an hour we had a fake wall. Now we could feel how light, heat, and airflow was affected. We discovered the idea was not a good one, and that it would be better to build the wall at the top of the staircase. That saved us both time and money.

The old dangerous path

What I have seen in many companies is that ideas are assessed (and sometimes suggested) by a senior person or committee. They user their experience to assess an idea. Simplified it looks like this:

Idea > Business Case > Senior Decision > Build > Measure success

I have worked as a management consultant, innovation lead, and product manager. The flow above is a common decision model to figure out if something is a good idea.

This decision process is sometimes done in committees, often through a stage-gate model. The model to the left is a stage-model I created and worked under in a +20.000 employee company. (I wasn’t any wiser back then.)

In this model, a committee has to decide (with their knowledge and assumptions) if an idea is good after an innovation team presents their desktop research.

There are several problems with this approach:

• We don’t know if the committee members have sufficient knowledge. And neither do they. It is all based on their assumptions and experiences. As markets move fast, their knowledge and assumptions might be obsolete.
• You appoint someone to work on an idea for some time before actually coming to kill knowledge (in the model, proof of concept is done after the business case is made). The result is that the people working on an idea often get attached to it, making it harder to kill.
• Working on a business case early in an idea assessment process is based on pure guesses. While it is beneficial to bear costs and benefits in the back of your mind, I have done many long and impressive business cases in my time to satisfy a need for numbers in a scenario about which we know almost nothing. For this I was praised. And (most of the time) I was wrong.

The experimental approach

The alternative to the old approach is to become experimental. In this approach, we try to run simple experiments as fast as possible in order to learn if something is a good idea. Here the focus is on testing fast. Simplified, the process would be more like:

Idea > Measure (experiment) > Senior decision > Build

Since experiments are run very early in the process, you need bright minds to come up with reliable ways to test. But within IT this is often quite easy.

For example: Is it a good idea to change the color of a button in our product to orange? Well, we can easily split test that on a website and figure out if more users click the orange button (see experiment on the left).

An important aspect of the experiment is that users should not know they are part of the experiment. I learned this from my time studying psychology; Psychologists always try to hide the true intentions of an experiment. The reasons are:

• People often do not behave the way they say they do.
• People change behavior if they know they are part of an experiment.

So an important aspect of the experimental approach is measuring behavior against your idea. If not, you might run into problems like the one described here.

Experiments are best when you measure behavior as long as users are not aware that they are part of an experiment.

When to include senior management

The experimental approach measures if your idea is good. That does not necessarily mean that it is good for your company. When you have run experiments and gained knowledge about an idea, this is the time to include senior management. Now they have real market data on which to base their decisions.

What does this mean for IT development?

The experimental approach has a number of implications for IT development:

• A development team’s responsibility is not only to deliver new features but also to run experiments to understand if something is a good idea; thereby identifying new opportunities.
• The way we measure performance should reflect the experimental approach: The focus of IT teams (and any other team) should be on outcome and not deliveries. If the team delivers something on time, budget and quality, but nobody wants to use it, we have wasted time and money.
• Experiments are not a one-time thing. When testing an idea, there are multiple aspects of an idea to test. Beginning with a simple experiment and expanding into multiple experiments means that you need to run experiments continuously.

All this text I’ve written already makes me tired. Show us the experiment! See them by digging into the experiments in our publication.