Validating Product Ideas: Think Before You Ship
When interviewing PMs and Product Designers, I ask candidates to walk me through a case study. As they walk me through their work I’m amazed at how often critical flaws are acknowledged but disregarded, and as a result, nonviable products are shipped. What amazes me more is that they are okay with it.
This appears to be a by product of a broad misunderstanding of the lean startup ethos. Plastered on start-up walls are slogans like “Ship & Learn,” and Facebook’s famous “move fast and break things,” which I’ve learned has changed to something more like “move fast but don’t bring the service down.”
It is true that some consumer apps can, with the luck of good timing, successfully take a “throw it at the wall and see what sticks” approach, but most are better served by taking a more thoughtful and perhaps less cavalier approach.
What this means in practice is applying more rigor to evidence-based decision making by taking a scientific approach to product lifecycle management. Paradoxically, spending more time upfront to prove out a product thesis increases the velocity of achieving the team’s, and by extension, the company’s, goals.
A team’s goals and the criteria by which they measure success should directly contribute, in straightforward and obvious ways, to the vision of the company, and the milestones it has set to realize that vision.
To be clear, I’m speaking of new products or major features. Once an idea moves into development and ships, rapid iteration cycles are indeed how one achieves product excellence. But only if success criteria are met, and progress towards a product’s intended goal is carefully monitored and analyzed. At the risk of stating the obvious, iteration cycles are about constant improvement and moving the needle in its intended direction.
I would argue that while this approach is necessary for mission critical applications, the core principles of the Scientific Method benefit any project, at any stage. Since assumptions are tested, refined and validated, the likelihood of achieving the intended outcome greatly increases.
Through the process of validating ideas, many don’t even make it to the development phase, and this is a good thing. It makes for hyper-focused products with a clarity of purpose, while freeing up resources to focus on fit-and-finish. All those seemingly trivial defects can be fixed on a compressed timeline, making for polished and high impact products that deeply resonate with their intended audience.
The Long Game
As is often the case with wildly popular manifestos (whether they be diet fads or business processes), the fundamental thesis becomes distorted over time, as practitioners seek to expand upon and push the ideas further.
Nothing has had a bigger impact on software development in the last decade than Eric Reis’s brilliant book “The Lean Start-Up.” In it, Reis describes practical methods for reducing the length of development cycles — essentially antidotes to the inertia experienced by software development teams that adhere to inflexible and long “waterfall” development cycles.
The problem Reis seeks to solve is the lack of experimentation, user feedback, and course corrections inherent in long development cycles. His theory is one of combining hypothesis-driven experimentation, iterative product releases, and what he calls “validated learning.”
It is here that Reis’s principles have been most mis-applied. Validated learning is a process in which “…one learns by trying out an initial idea and then measuring it to validate its effect.”
Many modern development teams have applied some variation of the agile scrum methodology to achieve a bi-weekly shipping cadence, most often referred to as a “sprint.” However many teams fail to put in place the goals, measurement, and analysis required to validate the effect of their efforts, thereby failing to reap the benefits of their adopted development process.
By deeply immersing themselves in the often rigid (scrum master), highly tactical (daily standups) and short sighted (two week outlooks) of agile scrum, they lose sight of the “why.” The big picture. The vision. The grand design.
Disruptive Innovation
While losing sight of a product’s ultimate purpose and the problems it seeks to solve is an unfortunate position for any technical team to find itself in, it is particularly harmful when one aims to change a deeply entrenched industry fundamentally.
Innovation theories, including diffusion of innovations and the work of Clayton Christensen and others around Disruptive Innovation, all share a common, base understanding: innovation, and especially that which is highly disruptive, takes a long time. In the words of Professor Christensen:
“Most every innovation — disruptive or not — begins life as a small-scale experiment. Disrupters tend to focus on getting the business model, rather than merely the product, just right. When they succeed, their movement from the fringe to the mainstream erodes first the incumbents’ market share and then their profitability. This process can take time… Complete substitution, if it comes at all, may take decades, because the incremental profit from staying with the old model for one more year trumps proposals to write off the assets in one stroke.”
In Diffusion of Innovation theory this phenomenon is often represented along an X (user types) and Y (market penetration) axis:
With successive groups of users adopting a new technology (shown in blue), its market share (yellow) will eventually reach dominance. In mathematics, the yellow curve is a logistic function:
- e = the natural logarithm base
- x0 = the x-value of the midpoint,
- L = the curve’s maximum value, and
- k = the steepness of the curve
Many of us are attracted to opportunities with the potential to completely upend existing industries.
As “disruptive innovators,” it is therefore imperative that we understand not just the next few legs of the journey, but the destination. In practical terms, this understanding will profoundly shape everything we do — from the tough compromises we make, to how we frame and consider user and industry problems and the innovative thinking we bring to bear in solving those problems.
To do so, we need to apply rigor to framing and considering the problems, understanding our users and validating assumptions and opinions.
The Scientific Product Method
PMs should approach product features with non-attachment and objectivity, applying a scientific method of evidence-based decision making. To provide context and help illustrate the benefits of this approach I’ve put together the following diagram.
There is nothing groundbreaking here, the Scientific Method was developed in the 17th century as a procedure consisting of systematic observation, measurement, and experimentations, and from that, the formulation, testing, and modification of hypotheses.
I have simply taken this approach and applied it to the specifics of software development.
- BASIC ASSUMPTIONS: Software design does not happen in a vacuum — an underlying, modern understanding of best practices and winning approaches is presupposed. Assumptions may be the outcome of a novel idea from a single individual or the result of a collective ideation exercise. Heuristics based on expertise and prior experience often weigh heavily, as do potentially misdirected cognitive biases.
- SOLUTION HYPOTHESIS: A proposed solution to a user need or problem based on information inputs. Per Occam’s Razor, when deciding among multiple ideas, the ones with the fewest assumptions should be selected for experimentation.
- INFORMATION INPUTS: Includes existing data and observable user behavior, client feedback, intuitive hunches, or the results of related studies or competitive products.
- DATA INPUTS: Quantifiable measurements and observations subjected to systematized statistical scrutiny.
- DISPROVEN HYPOTHESIS: If the results of studies do not support the hypothesis, it is important to understand why. A disproven solution hypothesis is often the result of incorrect assumptions about user motivation and behavior, or a biased misreading of the data. Also, new information and data inputs may contradict previously held views and assumptions, requiring modifications of, or outright rejection of the hypothesis. A disproved hypothesis, however, should not be misconstrued as the result of poor product work, it is simply a phase of the process. A failed test offers valuable insights that were previously unknown or misinterpreted. In any case, it is a step forward. The greatest benefit, however, is the prevention of wasted time and resources developing a misguided software solution.
- ITERATIONS AND IMPROVEMENT: When the data corroborates the thesis, and the team feels a high degree of confidence that the proposed solution will make an impact, it is important to predict the level of impact, and how much of an impact would constitute success. Acknowledging that technology and user behavior and needs are constantly evolving, a solution that serves that need must also continue to evolve. In addition to constant improvement, a by product of the rapid test, iterate and release cycle is that it often leads to the next big idea.
Variant Testing
There is general skepticism concerning variant (or A/B) testing, in that the studies often provide statistically insignificant, non-actionable results. The problem is not with the idea of variant testing itself, but the way in which they are executed, and the expectation that a single test will provide a clear answer.
Does Variant Testing Even Work?
Some quick research reveals that upwards of 80% of A/B tests are deemed “failures,” in that they don’t produce statistically significant signals to drive an evidence based decision. However, the problem here is not that variant testing doesn’t work. The problem is that most tests are scientifically flawed, or the testers fail to identify the often subtle signals revealed by the tests.
In my experience, the evidence produced by variant testing can have a significant impact on results. The trick is to run scientifically sound tests with a general understanding that a single variant test will seldom result in a binary answer to a tough question. Variant testing is about iteration and optimization. Achieving results is realized by running many experiments, identifying subtle signals in the results, applying the learnings, and carefully monitoring the results.
A test that results in inconclusive signals is not a failure. The signals can be used to design the next test. If a test is well designed (i.e. you aren’t running tests against the same control group on different days of the week or different times of day, or you aren’t running tests against varying control groups), it will tell you something, irrespective of its statistical significance.
The tests I am most skeptical of are those that result in profound results. Often I find that this is due to a biased reading of the data or a biased control group.
Understanding Basic Statistical Principles & Cognitive Bias
A/B testing relies on certain statistical principles that require a minimum sample size. When reading real-time results one often finds mysteriously large gaps between the variant groups, leading to premature conclusions and knee-jerk strategic pivots. But almost all anomalous, large fluctuations are the result of temporary imbalances between the groups. Over time these fluctuations reveal themselves as temporary outliers (that could be coincidental or caused by a multitude of factors that usually aren’t important).
As with any data set with abnormal spikes, the data should be normalized by removing the abnormal data. Or, if a pattern emerges from the spikes, further testing can reveal interesting, not yet considered contextual user behavior (e.g. the spikes are all happening on Sunday afternoons, so what is it about Sunday afternoons that is making our users behave this way?).
Cognitive bias is a tricky one that affects many aspects of product management. On the one hand, you want PMs to develop good product instincts based on their understanding of the user and the experience they’ve acquired.
On the other, previous experience can lead to a skewed reading of the data. Of even worse, an emotional attachment to an idea can lead to an irrational interpretation of the data, consciously or unconsciously. A good example of cognitive bias at play are the scientifically flawed books of Malcolm Gladwell.
Gladwell tends to come up with a provocative hypothesis and then prove the hypothesis by finding supporters of the idea and citing data that support it while ignoring contrarian points of view and academic research. Gladwell would make a bad PM.
Here are a few things PMs can do to help avoid the trap of cognitive bias:
- Ignore opinions and embrace evidence-based facts
- Be skeptical of “domain experts”; they often have an entrenched way of thinking
- Practice non-attachment to ideas, especially your own
- Grow attached to the problem, not the solution
