Lean Experiments
Three Methods For Setting Minimum Success Criteria
When running experiments on your business model it is important to identify the assumptions you want to test. However, most assumptions are vague and nonspecific. Such assumptions are not hypotheses until you set minimum success/fail criteria. The setting of minimum criteria is what turns an assumption statement into a falsifiable hypothesis. Hypotheses are specific and provide a basis upon which to decide whether or not the data supports our assumptions. Tristan Kromer has written a great post on this topic which I highly recommend.
Once the need for setting minimum success/fail criteria is understood, the next question is often about how to do it. I have found three methods that work which I will now describe below.
Early Adopters: When running experiments, innovators are often encouraged to focus on early adopters. If your team is running an experiment and they have explicitly decided to focus on a group of customers they are calling early adopters; then the minimum success/fail criteria should be very strict.
This is because early adopters are a specific set of customers who have the problem, are aware of having the problem, have been actively looking for a solution, have tried to create a solution of their own and have a budget. Early adopters are not a random sample of people. They are specifically selected using filter questions in order to identify them. As such, if you are running an experiment on a group you have defined as early adopters, your minimum success/fail criteria should be higher than 70% (i.e. 7 out 10 support your assumptions).
The reason for setting such a high bar is that we want innovators to think hard about who their early adopters are and then make the effort to identify them. If the experiment fails to meet the high bar we have set then we can have a conversation about who the early adopters for our product really are, or whether we are making the wrong assumptions about our early adopters’ true needs.
Industry Standards: Not all experiments will be run using early adopters. When you start setting up landing pages or engaging in public facing marketing and sales efforts, it becomes harder to identify early adopters in advance. In this case, a bit of research on successful business models within your arena or industry may be helpful. For example, the typical conversion rates for a B2B software as service company range from 3%-8%. Such a conversion rate can then form the basis for your minimum success criteria.
Back of the Envelope: Finally, it is also possible to do some back of the envelope maths. You can base these calculations on an evaluation of the success rates and price points you feel you need in order to have a successful business model. Before you make your calculations, it is important to get data that is as accurate as possible on costs, market size and possible price points. It is also important to note that you have to keep checking your initial calculations as you learn more, and make adjustments as needed.
These are the three methods that I have found work best, especially in the early stages. As your product becomes more established, you will have more reliable data about the various aspects of your business model that work well. Your minimum success criteria can be more informed by real data.
The list I have shared above is a not comprehensive list. Please feel free to share your own methods in the comments section below. The key is to make sure that you do not launch your experiments without first setting minimum success/fail criteria.
_______________________________
This article was first posted at www.tendayiviki.com.
