From Strategy to Sprint: Continuous Validation
Hello and welcome back to the Product Leadership Journal, where I write bi-weekly about managing products, people and technology. In my last article, I described my view of the Product Management Stack — an all-encompassing, big picture of all that goes into managing a product. As I progressed through my career, I found many resources describing best practices for a Product Owner, and on the other end, excellent guidance on evolving product vision & strategy. However, the part that I — and most product people I’ve worked with — struggled with the most is the middle of the stack. How do you connect a well-formed strategy to a healthy product development process? How do you connect the booster to the payload to get the proverbial rocket ship? 🚀
The answer lies in an effective and efficient Continuous Validation process, circled below. Like everything else in Product Management, building the skillset requires building a) the mindset and b) the toolset. Each of these is described in its own section below.
The Mindset
These are some ideas that I consider foundational to establishing the discipline of Continuous Validation.
1. Fail fast — or don’t
Contrary to what Silicon Valley culture promotes, you don’t need to “fail fast”. It’s totally fine to not fail at all. Success is not as elusive, or as expensive, as you might think. Lots of companies get it right through a disciplined, frugal innovation process. For example, Meta lost (“invested”) $10 billion in R&D last year, which equals the total private-sector funding in space-related companies for the same period, SpaceX included[1]. It’s all about constraints that the leadership/business/market/team is willing to impose. Innovation is truly valuable only when it’s exponential.
2. All costs are relative
Continuing the above analogy, NASA spent $280 billion in today’s money to send the first humans to the moon. Ridiculous, right? But the (perceived) Cost of Delay was higher[2][3]. The key is to draw a clear distinction between building to learn and building to earn[4]. Generally speaking, product discovery experiments are not revenue-generating product deliverables — yet we have a natural tendency to kill two birds with one stone (and get 20 bugs in return). I’ve personally witnessed the loss of millions of dollars simply because the first usable prototype was misunderstood — internally and externally — to be the final deliverable. (Fortunately we were able to turn things around and even partner with some of the affected customers to accelerate early deployments).
3. Watch out for zombie features
You’ve probably heard this before: no one likes to be told their baby is ugly. Misaligned or unviable ideas need to be eliminated before their creators get too attached to them (or better still, before they even create them). Otherwise, we get zombie features or worse, zombie products. In the previous example, the team demo’d a mocked-up UI with fake data, and it was too late before everyone reluctantly accepted that it had no soul (backend or architecture).
4. Mind your language
Product Leaders need to put in the hard work to get everyone in the organization, from interns to executives, comfortable speaking the same language. The key is to abstract more and simplify less. I learnt this the hard way. For months I thought I was doing an outstanding job of translating technical terms into business language, only to discover that lack of a common vocabulary was making it difficult for stakeholders to grasp the development team’s progress & challenges. I created alignment by building awareness, reinforcing it at every opportunity, and coaching. In some cases this meant getting the business to understand a common technical term in plain words (e.g. “an art asset is a 3D object that can be placed in the scene, such as furniture”), in others it meant adopting market terminology in technical discussions (e.g. “in situ” instead of “on premise”). This resulted in a shared view of the product, which also improved the effectiveness of OKRs (win-win).
5. Single source of truth
I strongly discourage management by PowerPoint wherever I see it. I prefer dashboards, reports & summaries that are semi/automatically generated from real data, ideally on demand. It takes time and effort to setup and maintain, but there are excellent tools for this now, and very little reason to paint a pretty picture by hand. Together with a shared vision, shared vocabulary and continuous visible progress, it should be fairly easy to whip up different views of the same data depending on who the audience is. This is also where a ProductOps team can shine[5].
6. Know thyself
In Product Management we talk about doing (researching, building, validating) just enough to reduce risk for the next stage or iteration (See Product Kata[6]). But how much is enough? I like the idea of being in the “Goldilocks Zone”: the right balance between confidence and doubt.
The Toolset
Now that we’ve discussed the mindset, let’s talk about some concrete tools to apply the Continuous Validation process in practice.
1. Confidence Dashboard
Product Managers get tons of input on what should be built. Stakeholder input, user research, user feedback, customer requests, internal ideas, inspiration from the outside world. Collecting and prioritizing them is a big part of the job. It’s important that everyone feels heard. It’s equally important to ensure that the focus is on the right thing — whether it is to build or research. So imagine all of these inputs going down a funnel, and only those that make it beyond a threshold % of confidence making it all the way down, where time & effort are spent on them.
The filtering could be highly structured, e.g. through regular product initiative meetings, or ad-hoc & fuzzy, based on the nature of the organization & industry. Even if you don’t adopt a solid framework for it, it helps to ask the question: do we have enough confidence to invest resources in this thing yet? If the answer is no, either focus on something else or take the necessary steps to improve confidence (e.g. user or market research).
2. Build, Buy or Partner Scoring Framework
Once you’ve established it’s worth spending resources on, the next question is: what kind of resources? From a lean innovation standpoint, you want to maximize RoI and if you must fail, fail as early and cheaply as possible. Maximize bang for the buck and minimize the bang. There is a huge cost to maintaining anything that’s built[7], especially software.
Ensure that a rigorous Build vs Buy vs Partner analysis is built into your process. I created my own framework for this inspired by this Product School talk[8], which in turn draws on work done by others.
“The world is full of fascinating problems waiting to be solved. No one should ever have to solve a problem twice”. If someone has already solved it, reuse it, adapt it or integrate it. That’s how you get the exponential innovation curve. You can always swap it out for your own implementation in future if needed. Your product should be engineered to allow that, anyway.
3. Make room for user feedback
Quick: Build the next thing in the roadmap, fix bugs in the previous release or incorporate the outcomes of new feedback received from users? The right decision can make or break the product, or even the company. User feedback — whether in the form of qualitative usability testing or interviews, or quantitative A/B testing, analytics or surveys — must find its way into the backlog. Often this feedback is competing for resources with other commitments and priorities. It can’t be an afterthought, nor can it be the only thought. Sometimes there are good reasons not to incorporate it, or push it to a future release. But it is important to communicate this well, internally and externally.
A clear vision and strategy are already half the job done, because they provide vital context. For the rest, Product Owners need to plan sprints in a way that there is room to explore, analyze and prioritize user feedback. Product Managers need to ensure that it is acted upon (either implemented or communicated, and communicated fast if it doesn’t align with existing hypotheses). Note that here I’m talking about evaluative user research, i.e. validating the solution, not generative user research, i.e. finding the problem.
4. Deploy fast, rollback faster
I often quote this Airbnb Engineering Culture article[9]. Back in 2014, they were deploying to production 10 or more times a day, with each deployment taking under 8 minutes. But the clincher is that rolling back changes is equally quick and hassle-free. These are measurable metrics for your DevOps pipeline.
Equally important is for product architecture to enable features to be developed in a way that they can be exposed, hidden, rolled back and when necessary killed off without impacting anything else. In my experience, this is a hugely underestimated skill especially when it comes to complex software, such as platforms. Fortunately, there’s a book for that[10].
Final Thoughts: Product Values
One last thing that I’ve found impactful: a one-pager on Product Values agreed upon between product management, design and engineering. Typically this would come out of a workshop and codify trade-offs to make everyday decisions easier for everyone working on the product. For example, in our product we value “Consistency over customizability”. Fundamental agreements like these are highly effective at creating alignment and filtering out distractions.
I hope you found this useful. The cheapest way to fail fast is to learn from someone else’s mistakes 😉. Please leave a comment if you’d like to add more or hear more, and hit Follow if you found this useful. I’m also happy to connect on Twitter or LinkedIn.
References/Further Reading:
- SpaceX in all of its existence has raised $7.7 billion, significantly less than Meta’s “investment” in 2021.
- Cost of Delay at Black Swan Farming
- WSJF (Cost of Delay / Job Size) at SAFe
- “Escaping the Build Trap” by Melissa Perri
- “Why Product Operations is set to be the Backbone of Product-led Growth” at Mind the Product
- The Product Kata by Melissa Perri
- “Innovation is overvalued, maintenance often matters more”. “Hail the Maintainers” at Aeon.
- “Webinar: Build vs Partner — How to Choose by former PayPal Product Leader” hosted by Product School
- “Engineering Culture at Airbnb” published in the Airbnb Tech Blog
- “Just Enough Software Architecture” by George Fairbanks
