Running Experiments to Learn Quickly
In order to reduce the unknown unknowns, you want to iterate through well-designed experiments as rapidly as possible. Usually, designing an experiment doesn’t take much effort. It just takes some thought.
Emily Rosa, as a nine-year old girl, singlehandedly disproved that touch therapy can work as a form of alternative medicine. In fact, her findings were published in the Journal of the American Medical Association, the most prestigious peer-reviewed medical journal in the US. She also landed in the Guinness Book of World Records, because she was so young as a researcher.
She accomplished this by putting touch therapy practitioners behind a cardboard screen. They put their hands through holes at the bottom of the cardboard. These of course, were the hands which they claimed helped with healing. Then she held her own hand above theirs (either left or right). The touch therapists then guessed where she was holding her hand.
Their accuracy? 44%. Worse than random.
Notice that executing the test was quite simple. All she needed was basic knowledge of statistics like mean and probability. It just required a bit of ingenuity on her part, to construct a test which would be valid. She took advantage of the fact that she was so young, as the therapists had their guard down, not expecting a nine year old to be a threat.
Quite often the type of testing required doesn’t even require that level of sophistication. Sometimes it’s enough to just take a handful of spaghetti, throw it against the wall, and see what sticks. If there are a number of possible approaches, often a quick test identifies that a particular approach is not worthwhile. If you have a number of unknowns, you may be able to truncate the list of options to something much shorter with an approach similar to the sticky spaghetti approach.
Google Adwords or Bing Ads are a simple way of executing such a spaghetti approach for many types of products. By writing a handful of ads, one for each product type or feature, and directing each one to a relevant keyword or group of keywords, you simulate a real buying situation, without having the product. As users click on your ads, you get statistics on customer responsiveness. This approach, where you check whether or not customers actually respond (as opposed to just say they will), gives you a high fidelity signal of your prospects’ intentions. You can rapidly rule out a number of potential products, if your expected customers barely respond to your ad.
This approach was famously used by Tim Ferriss when he chose the title of his first book the 4 Hour Work Week:
Ferriss decided to look for some data. He took 6 prospective titles
that everyone could live with: including ‘Broadband and White
Sand’, ‘Millionaire Chameleon’ and ‘The 4-Hour Workweek’ and
developed an Google Adwords campaign for each. He bid on
keywords related to the book’s content including ‘401k’ and
‘language learning’: when those keywords formed part of
someone’s search on Google the prospective title popped up as a
headline and the advertisement text would be the subtitle. Ferriss
was interested to see which of the sponsored links would be
clicked on most, knowing that he needed his title to compete with
over 200,000 books published in the US each year. At the end of
the week, for less than $200 he knew that “The 4-Hour
Workweek” had the best click-through rate by far and he went
with that title.
[source: thulme.com]
By using a search engine like a shop window on a busy street, you get insight into consumer preferences at a very low cost. Moreover, you get statistical significance quite quickly, so that you can achieve much greater confidence in your product idea before committing to it strategically. Also, you get back information very rapidly for popular keywords, so you can turnaround on a number of similar tests without really committing time and effort to creating a product. This can easily be estimated in a spreadsheet, once you do one such test. If you have multiple hypotheses you want to test, you will know roughly when you can start building out your product further.
If you liked this post, go to Launch Tomorrow ~ Landing Pages for your Lean Startup for more.
