Principles of Experiment Flow
Running experiments for your product and business can be overwhelming, but there are ways to manage the process by following three simple principles.
I’ve been a fan of agile and lean principles since the late 1990’s, but it wasn’t until the third startup I had joined was failing that it dawned on me…
These principles were great for delivery, but what about discovery?
I don’t want to efficiently build things that nobody wants. Why can’t we pull these principles for delivery up into our discovery process?
That’s when I stumbled into the business model generation and lean startup movements.
Since then, I’ve been working with hundreds of teams all around the world. All of them struggling with making experimentation and discovery a repeatable process, not just an initial phase before delivery.
Based on what I’ve learned applying lean concepts to experimentation, here are a few principles I’ve found to help you get your experiments flowing.
Principle #1 — Visualize Your Experiments
I’ve found inspiration from the lean and kanban movements particularly on this principle. If you keep this work all in your head, then you’ll never be able to achieve flow. Not only are you teammates unable to read your mind, so much of flow requires you to visualize your work.
Step 1 — Write down your experiments
I’d recommend only one experiment per sticky, to keep things organized. You don’t have to write down 100’s of experiments, only the ones you feel you’ll be running over the upcoming weeks.
Step 2 — Draw a simple experiment board
The columns from left to right are:
- To Do = Experiments that are on your list to do
- Doing = Experiments that are in progress
- Done = Experiments that are no longer running
- Analysis = Experiments that are being analyzed to extract insights
I’d sit down with your team and have a conversation about each column, so it is clear what each column means, otherwise it’ll cause confusion later on.
This is one of the simplest forms of an experiment board you can create. I’d recommend starting here before including additional columns.
For reference, I’ve been playing with this format for quite some time and used to like the “Validated” column, which I got originally from Eric Ries. Over time, I’ve started to back off a bit on that language because teams will set the bar so low on their hypotheses that they’ll artificially validate them and move on too quickly.
Step 3 — Add your experiments to the To Do column
Rank your experiments from top to bottom, where the top is the one you are going to do next.
Step 4 — Pull the experiments across the board
As you begin to write the customer interview script, pull the customer interviews sticky over to the Doing column.
When you’ve finished with this round of customer interviews, then pull it over to the Done column.
And when you’re analyzing the interview notes to extract themes and patterns to inform your next experiment, pull it over to the Analysis column.
In this example above, what you’ve learned running 15–20 customer interviews informed your survey design as your next experiment on the To Do list.
Principle #2 — Limit Experiments in Progress
Ok so visualizing your experiments sounds simple enough right? Why would you even need more principles? Well something I’ve learned with kanban and team behaviors initially surprised me.
This is what teams normally do without the guiding principle #2.
Teams inherently underestimate how much work it is to run experiments, especially if they’ve never ran them before. So it should come as no surprise, that they pull all the experiments over at once and try to do them all in parallel. This results in slowing the entire process down. It’s also difficult to extract insights from a previous experiment to inform your next one.
Step 5 — Set work in progress limits for your experiments.
Let’s say you start with a limit of 2 on your Doing column.
This will prevent the team from pulling a 3rd experiment over until 1 of the other 2 is moved into the Done column.
This is important if you are to ever achieve experiment flow, so your board looks ideally something more like this…
In the example above, the team synthesized the findings from the customer interviews to inform the survey design. As the survey was wrapping up, they started on their online ads to drive traffic to a landing page. The experiments are flowing, using what’ you’ve learned to inform your next experiment.
Principles #3 — Continuous Experimentation
The last principle, which also finds its roots in lean and kanban, is the idea of continuous experimentation. A team that starts with the board above to achieve flow will eventually outgrow it. You don’t want the board to artificially constrain the team from growing and maturing over time.
I recommend every couple weeks, having a retrospective or kaizen with your team to discuss:
- What’s Going Well?
- What Needs Improved?
- What Do We Want to Try?
Those three questions will inevitably lead to experiment board suggestions.
Let’s say the team is trying to line up customer interviews, but the user research department won’t let you do so. They state that it’s against company policy to talk to customers. Well that would be a “blocker” that’s preventing you from making progress on that experiment.
It’s a good idea to identify and visualize these, which will help you communicate progress to stakeholders on why things are slowing down. It’s hard to achieve flow when you are always blocked.
Splitting Columns Experiment
Maybe the team has outgrown the initial board and is frustrated that the Doing column doesn’t capture the nuances of experimentation.
There is work developing the experiment but then you have to run it, which also takes time. When we talk about the board, it would be great to see which ones are still being developed vs which ones are running.
There are countless examples of these that your team will experiment with over time to improve flow.
Some will work.
You can always revert back the ones that don’t work and keep the ones that do, since you are revisiting this every couple weeks.
Running one experiment is hard enough, but making it a repeatable process is what will help you leap frog competitors in the long run. By following these three simple principles, I’m certain they will help you and your teams get your experiments flowing in no time.
If you would like to learn more, Alex Osterwalder and I are writing a book together on how to test your business ideas. Go sign up to be notified when it launches.