by Dan Toma & Tendayi Viki (authors of The Corporate Startup)
As a dashboard is important for a pilot to be able to navigate and get a accurate understanding of the current situation, a dashboard like structure is useful for corporate scientists to get an understanding of what’s happening with the ongoing experiments.
The following Experiment Canvas is based on the Experiment Map designed by Brant Cooper, which I personally have played around with for the past 2 months. Using the tool, put forward by Brant, in my daily activity I have identified some limitations — limitations that have been addressed in the Experiment Board.
So let me talk you through.
Scope of the board. As with Brant’s work, the scope stays the same, recording business experiments over a period of time.
Who is it for. Entrepreneurs, intraprenrus — in a broader context, corporate scientists.
Steps in using it:
#1 Like in the case of hypothesis driven development, start by identifying and stating the leap of faith assumption.
We assume that providing push notification will increase the overall retention rate of our app.
#2 A potential validation of a business assumption influences in one way or the another the entire business model. So the second step in using the Experiment Board is to gray out the business model canvas block impacted the most by the ongoing experiment. Bare in mind that all business model canvas blocks are interconnected, so altering one can and will result in changes in other block as well.
#3 Not all experiments address the entire audience of your business. You need to be specific about the group which is going to be exposed to the experiment.
We assume that providing push notification will increase the overall retention rate of our app on Android devices.
The cohort can be anything from a particular geography (people in Kansas), to a particular demography (male, female) or any other characteristic that they share: lifestyle, devices that they use, operating systems, ethnicity etc.
#4 Design the experiment. How exactly is the experiment going to look like, are you going to do a fake door, a landing page, a paper prototype etc. Remember though that this is still an experiment, so you don’t need to go gang busters building the entire set of features, you just need to use the minimum set of requirements providing the most learnings.
#5 Like in the case of Alice in Wonderland, if you don’t know where you want to go any road will take you there — specify the success criteria of your experiment (when is the tested assumption going to be validated?).
For this assumption to hold water, the increase in the overall retention rate should exceed 32%.
For this step it is important to have relevant success criteria — don’t lower your standards just because you want the assumption to be validated, cause if you do so, somewhere down the line you’ll find out that you just became successful in failing.
#6 Like in design thinking, the experiment needs to be time boxed — it’s a big difference between getting 10,000 people validating your assumption in 1 day and the same number in 10 months. So as you can see, although the numerical value is the same, the associated temporal element plays an important role.
For the experiment to succeed, the tested assumption needs to be validated within 4 weeks after the launch of the experiment on February 12th.
Time boxing increases the specificity of the success criteria hence making the outcomes of the experiment more relevant.
#7 After running the experiment for the duration of time specified in the time box field you need to start collecting and aggregating the data.
#8 Conclusion are drawn (lessons learned) by interpreting the data collected in the previous step. Insight is generated when actual test results are compared to the expected ones.
Based on the gather numbers we have the following insight in our initial assumption: _____________
#9 There is no point in running experiments if at the end of the exercise you are not able to have a decision with respect to the initial assumption.
Write down the decision you took at the end of the experiment, be it a new iteration of the same experiment based on the newly gathered insight or an ‘invalidated, move to the next assumption’ one. As a rule of thumb it’s better to try to triangulate insight before concluding that a particular assumption has been invalidated — and by triangulation I mean having at least one more data set that will be able to confirm the findings of the initial experiment.
Before initiating an experiment remember 2 things: first of all is more important to manage the cost of failure rather than the rate of failure and second, according to Thomas A. Edison, you don’t fail you just find ways in which it doesn’t work.
