Stop Validating and Start Falsifying
The product management and startup worlds are buzzing about the importance of “validation”. In this entry, I’ll explain how this idea originated and why it’s leading organizations astray.
Why Validate?
In lean startup circles, you constantly hear about “validated learning” and “validating” product ideas:
The assumption is that you have a great product idea and seek validation from customers before expending vast resources to build and bring it to market.
Indeed, it makes sense to transcend conventional approaches to making product decisions. Intuition, sales anecdotes, feature requests from customers, backward industry thinking, and spreadsheets don’t form the basis for sound product decisions. Incorporating lean startup concepts, and a more scientific approach to learning markets, is undoubtedly a sounder approach.
Moreover, in larger organizations, sometimes further in the product life-cycle, everyone seems to have an opinion about such aspects of the business model as:
- The most pervasive and urgent market problems the product should solve.
- Whether the solution truly addresses the market problems.
- What price customers would be willing to pay.
- What tactics will most effectively drive prospects to buy.
Presumably, the “validated learning” approach of lean startup enables the organization to identify which opinions are factual and not mere speculation.
Validation in Practice
But let’s consider what our naive lean startup practitioner (we’ll call her “Cameron”) does in practice.
Present the idea
Cameron puts together a slide deck, minimum viable product (MVP), or demo, presents her idea to one or more prospects, and eagerly awaits feedback. The prospects say, “I like it!” or “I would buy it!” Cameron feels warm inside, as the prospects have “validated” her and her idea.
Ask about the “pain point”
Since solving an urgent and pervasive problem or “pain point” is usually a prerequisite for product success, Cameron visits with prospects and asks them if they experience one of the problems she envisions her product would solve. “Yes,” reply the prospects. Feeling “validated”, Cameron does an internal high-five with herself.
Name a price
Cameron asks prospects what they would pay for a solution with the features she envisions for her product. Prospects “name their price”. Or, in equally naive fashion, Cameron herself names a price and asks, “Would you pay X for the product?” Prospects reply affirmatively. Cameron can hardly contain her giddyness as her pricing assumptions are “validated”.
Cameron feels extra special for having “gotten out of the building” to visit customers.
What’s Naive about Cameron’s Approach?
The results of all of Cameron’s efforts are practically worthless, aside from the emotional affirmation she may feel. When you visit prospects, you don’t get reliable information by posing hypothetical questions. When you seek validation from someone, you will tend to get it. Cameron unwittingly designed her interactions with prospects in a manner likely to confirm her preconceived ideas, and her interpretations of the results were naive. Let’s call it “validation bias”, an insidious psychological disorder that has infected the lean startup community.
Note: had Cameron conducted a survey, though seductive because it would have yielded quantitative data, the results would still have been suspect to the extent the questions were hypothetical and failed to confront prospects with real choices and commitments.
What’s the Alternative?
If you want to be scientific in your approach to product decisions, you craft experiments and make falsifiable predictions, with the intention of testing, not “validating”, the hypotheses and underlying assumptions.
Philosopher Karl Popper is famous for having championed falsificationism, a set of principles that shifted the emphasis from verifying scientific beliefs to ensuring they are possible to falsify via experiments:
According to Wikipedia:
“A statement is called falsifiable if it is possible to conceive an observation or an argument which proves the statement in question to be false.”
In fact, Popper argued that a statement isn’t scientifically factual at all if it isn’t falsifiable.
What Cameron Could Do Differently
Instead of asking hypothetical questions, Cameron could ask the prospects what they actually do (and have done) instead of what they would do. She could probe into their current and past experiences and note whether the supposed problems manifested themselves in those experiences. She could sit with prospects in their native environments and observe (ethnography), first hand, their situations and challenges. Cameron could test her value proposition and pricing hypotheses by quantifying the costs that problems and their workarounds impose on prospects. She could ask for an actual commitment to pay for the product once it’s released.
Cameron could also work with her team to craft “digital” experiments and predict how those experiments will turn out. For example, her team could author “how-to” guides, downloadable on landing pages, that help prospects solve the problems she assumes they face. The team could predict how many people will visit those landing pages and how many people will proceed to download the guides. (To make prospects aware of the how-to guides, the team could run Facebook ads or Google AdWords to drive people to those landing pages and see if the predicted number of page visits and downloads materializes.)
The possibilities are endless, but they require a mindset that emphasizes falsifying product ideas and business model hypotheses, not “validating” them.
Originally published at blog.cauvin.org on February 9, 2014.
