Principles of Experiment Flow

David J Bland
Feb 26, 2019 · 6 min read

Running experiments for your product and business can be overwhelming, but there are ways to manage the process by following three simple principles.

Image for post
Image for post

I’ve been a fan of agile and lean principles since the late 1990’s, but it wasn’t until the third startup I had joined was failing that it dawned on me…

These principles were great for delivery, but what about discovery?

I don’t want to efficiently build things that nobody wants. Why can’t we pull these principles for delivery up into our discovery process?

That’s when I stumbled into the business model generation and lean startup movements.

Since then, I’ve been working with hundreds of teams all around the world. All of them struggling with making experimentation and discovery a repeatable process, not just an initial phase before delivery.

Based on what I’ve learned applying lean concepts to experimentation, here are a few principles I’ve found to help you get your experiments flowing.

Principle #1 — Visualize Your Experiments

I’ve found inspiration from the lean and kanban movements particularly on this principle. If you keep this work all in your head, then you’ll never be able to achieve flow. Not only are you teammates unable to read your mind, so much of flow requires you to visualize your work.

Step 1 — Write down your experiments

Image for post
Image for post
Get the experiments out of your head and written down.

I’d recommend only one experiment per sticky, to keep things organized. You don’t have to write down 100’s of experiments, only the ones you feel you’ll be running over the upcoming weeks.

Step 2 — Draw a simple experiment board

The columns from left to right are:

  • To Do = Experiments that are on your list to do
Image for post
Image for post
A simple experiment board

I’d sit down with your team and have a conversation about each column, so it is clear what each column means, otherwise it’ll cause confusion later on.

This is one of the simplest forms of an experiment board you can create. I’d recommend starting here before including additional columns.

For reference, I’ve been playing with this format for quite some time and used to like the “Validated” column, which I got originally from Eric Ries. Over time, I’ve started to back off a bit on that language because teams will set the bar so low on their hypotheses that they’ll artificially validate them and move on too quickly.

Step 3 — Add your experiments to the To Do column

Rank your experiments from top to bottom, where the top is the one you are going to do next.

Image for post
Image for post
Place your experiments in the To Do column

Step 4 — Pull the experiments across the board

As you begin to write the customer interview script, pull the customer interviews sticky over to the Doing column.

Image for post
Image for post
Pull the next top experiment over

When you’ve finished with this round of customer interviews, then pull it over to the Done column.

Image for post
Image for post

And when you’re analyzing the interview notes to extract themes and patterns to inform your next experiment, pull it over to the Analysis column.

Image for post
Image for post

In this example above, what you’ve learned running 15–20 customer interviews informed your survey design as your next experiment on the To Do list.

Principle #2 — Limit Experiments in Progress

Ok so visualizing your experiments sounds simple enough right? Why would you even need more principles? Well something I’ve learned with kanban and team behaviors initially surprised me.

This is what teams normally do without the guiding principle #2.

Image for post
Image for post
Don’t do this.

Teams inherently underestimate how much work it is to run experiments, especially if they’ve never ran them before. So it should come as no surprise, that they pull all the experiments over at once and try to do them all in parallel. This results in slowing the entire process down. It’s also difficult to extract insights from a previous experiment to inform your next one.

Step 5 — Set work in progress limits for your experiments.

Let’s say you start with a limit of 2 on your Doing column.

Image for post
Image for post

This will prevent the team from pulling a 3rd experiment over until 1 of the other 2 is moved into the Done column.

Image for post
Image for post

This is important if you are to ever achieve experiment flow, so your board looks ideally something more like this…

Image for post
Image for post

In the example above, the team synthesized the findings from the customer interviews to inform the survey design. As the survey was wrapping up, they started on their online ads to drive traffic to a landing page. The experiments are flowing, using what’ you’ve learned to inform your next experiment.

Principles #3 — Continuous Experimentation

The last principle, which also finds its roots in lean and kanban, is the idea of continuous experimentation. A team that starts with the board above to achieve flow will eventually outgrow it. You don’t want the board to artificially constrain the team from growing and maturing over time.

I recommend every couple weeks, having a retrospective or kaizen with your team to discuss:

  • What’s Going Well?

Those three questions will inevitably lead to experiment board suggestions.

Blocker Experiment

Let’s say the team is trying to line up customer interviews, but the user research department won’t let you do so. They state that it’s against company policy to talk to customers. Well that would be a “blocker” that’s preventing you from making progress on that experiment.

Image for post
Image for post
Tagged blocked experiments with a red sticker

It’s a good idea to identify and visualize these, which will help you communicate progress to stakeholders on why things are slowing down. It’s hard to achieve flow when you are always blocked.

Splitting Columns Experiment

Maybe the team has outgrown the initial board and is frustrated that the Doing column doesn’t capture the nuances of experimentation.

There is work developing the experiment but then you have to run it, which also takes time. When we talk about the board, it would be great to see which ones are still being developed vs which ones are running.

Image for post
Image for post
Example of column splitting

There are countless examples of these that your team will experiment with over time to improve flow.

Some will work.

Some won’t.

You can always revert back the ones that don’t work and keep the ones that do, since you are revisiting this every couple weeks.

Running one experiment is hard enough, but making it a repeatable process is what will help you leap frog competitors in the long run. By following these three simple principles, I’m certain they will help you and your teams get your experiments flowing in no time.

If you would like to learn more, Alex Osterwalder and I are writing a book together on how to test your business ideas. Go sign up to be notified when it launches.

Precoil

Lean Startup, Design Thinking & Business Model Innovation

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store