An Introduction to Experiment Pairing
How to build upon a previous experiment, without throwing it all away.
Congratulations you successfully ran an experiment. It’s a big accomplishment and should be celebrated. Not every team gets the chance to challenge the assumptions in what they are building.
But the excitement is short lived and now you are faced with new questions:
- How do we use what we’ve just learned?
- What do we want to try next?
- Do we have to throw all of this away?
Many experiments I observe with teams do end up being thrown away, whether it is physical or digital. However some show signs of promise. Yet teams struggle with how to build upon the previous experiment.
This is where teams can benefit from what I’m calling Experiment Pairing.
Experiment Pairing is simply knowing what other techniques fit with an experiment to further advance your learning.
Let’s start with a Landing Page. It’s one of the most common experiments I see and somehow the most under utilized when it comes to learning potential.
Once you’ve created a Landing Page, you are going to quickly realize that people aren’t going to magically find it. You’ll have to drive traffic to the page in order to gather data. This is where Online Ads come in handy.
Now you have targeted online traffic coming into a landing page and people are signing up. Awesome, but what do you do after that?
Well hopefully you have their email, so you should reach out to them.
When you email those who signed up, you should have an idea about what you are trying to learn. You already know that they signed up (the “what”) but you don’t know “why” they signed up. There are a couple different ways you can get that learning from customers through email, such as Surveys and Customer Interviews.
Let’s say that the Surveys and Customer Interviews went well and people are thrilled about your value proposition. Great, but that’s still rather weak evidence, because you’ve not delivered value and what people say can be different than what they do. At the end of your Customer Interview or Survey, I suggest having a “May we contact you in the future?” line so that you can continue to test with them. That testing often takes the form of a Clickable Prototype (interactive but not fully built) or Concierge experiment (where they see you manually deliver value).
If that went well, then you can do things like replace the call to action for customers to try the service. Then pair the Landing Page with a Wizard of Oz experiment, to manually deliver the value directly to a customer. This uses your Landing Page as a sort of digital curtain to manually deliver value to an end customer. It removes the bias of people witnessing a person delivering the value directly to them.
This moves you from weaker evidence (Online Ads and Landing page) to stronger evidence (Wizard of Oz). It also advances your learning as a team.
The Wizard of Oz is produces stronger evidence because there is a real value exchange occurring:
- You are learning if the customer accepts the value provided (desirability)
- You are learning what it costs to provide the value manually (viability)
- You are learning the steps needed to provide the value (feasibility)
And if you didn’t choose these experiment pairings in this order, then that’s ok too.
For example, perhaps you didn’t start with a Landing Page at all and decided on Customer Interviews. Then you can use what you’ve learned in Customer Interviews as the text in your Online Ads and Landing Page.
There are many, many different experiment pairings available to you.
I used a Landing Page for these examples but really, it’s only the tip of the iceberg. In B2B you can use Physical Brochures instead of a Landing Page. With physical products a Clickable Prototype isn’t applicable but you can quickly 3D Print a prototype instead. You have multiple experiment pairings available depending on your specific context.
A note of caution. Do not immediately stack several experiments together to address all of your risk in one shot. It will take you forever to get the experiments out the door. Instead, incrementally increase the strength of evidence with smaller, integrated experiments over time so that you know you are on the right track.
I’m confident that once you see how your experiments pair together, it will become your super power to rapidly reduce uncertainty.
If you would like to learn more, Alex Osterwalder and I are writing a book together on how to test your business ideas. Go sign up to be notified when it launches.