ROI Can’t Be Your Only Prioritization Tool (And five frameworks for making decisions steeped in ambiguity)
Setting the Stage
If you are a product person, I’m sure you can relate to being in a kickoff for a new initiative. Looking at a long list of desired features, a stakeholder in this session will turn to you and ask, “Ok, well, how do we get these all into the backlog?” and you will respond, indubitably, “We need to prioritize it.”
If this follows the pattern of dozens of sessions I have been part of, a backlog gets created, high-level epics written, dependencies captured, and effort swagged. Then a long list of features will be out there for everyone to review, and you will never have the time, budget, and resources to land it all.
Then, the elephant enters the room as you go through each line item. You will come to a table stakes feature for your customer, and you believe, no, you know deep in your being, it is a must-have. That stakeholder will then ask, “So, what is this ROI?” You won’t have a good answer.
Why ROI is Often Problematic
How does one prioritize the delightfulness of a product? Customer Feedback Interviews and prototyping tools ensure you have the correct product solution pre-launch. And post-launch, you can take NPS Scoring to measure customer satisfaction. But how does the person in charge of the purse strings know when to pull the trigger to execute the build of a feature?
In a fantastic post from 2008 — Luke Hohnmann wrote a long list of reasons that ROI prioritization is problematic, and this thought spoke to me deeply:
“To establish a realistic ROI, you need to engage in appropriate market research. Unfortunately, most market research that can establish realistic ROI, such as a conjoint analysis or a willingness-to-pay study, is usually based on relatively large chunks of functionality that may or may not correlate to what is on the backlog. The unfortunate side-effect is the release planning process, in which the development team negotiates the actual features with product management, can result in changes to the tested features that are so large the original research is invalidated.”
- Luke Hohnmann
All the metrics in the world can’t help us answer this question 100%, so we need to use agile methodology to get our best framework for prioritization.
Making Decisions in Ambiguity
Making decisions is complex and expensive. But not having an answer means fear, uncertainty, and doubt are sure to creep in, especially if you are paying someone like me, a product consultant, for our time.
Day One Hypothesis
One of the best things we can do (and thanks to Shobhit Chugh for sharing this framework) is the Day One Hypothesis.
“Without having all of the data — and you will never have all the data — you can’t be expected to forecast future value with a lot of precision. So, strive to be directionally right, and the same order of magnitude.”
Ameet Ranadive
Initially, a term coined by McKinsey stresses the belief that project work is a constant state of iteration. Nothing that is planned out by the minute will ever be right.
Instead, the idea is to speedily get to a decision that parties agree is of quality. The challenge is to get to a hypothesis to allow everyone to start executing but learning as much as possible. The three tenants of this process:
- Have the correct stakeholders. Less hierarchy and having the decision-makers in the room will lead to decisive actions. Remove as much bureaucracy and reveal “real” problems.
- Align with Company Strategy. Build towards something in which the company already has expertise — being a commerce company and building a robot are probably not excellent alignments.
- Commit. Get buy-in from everyone. The biggest challenge of being a PdM is building consensus. If you leave the room with all parties in agreement, many more doors will open.
There is less dogma of the heuristic exercises or methodology you use to get to these outcomes, but a full court press on speed to decision. Some of the GV Design Sprint picks up on defining a different framework for this concept.
MoSCoW Method
Suppose the consensus is that stakeholders will not commit to building something until they have a more concrete understanding of the upside. In that case, the MoSCoW framework might be a great middle-ground solution.
This prioritization method was developed by Dai Clegg in 1994 for use in rapid application development (RAD).
The system quickly buckets work into four levels:
- Must Have are requirements that need to be delivered, or the project will be considered a failure.
- Should Have are things that are just as important, but there might be workarounds to satisfy the functionality or can roll out incrementally.
- Could Have are things that would be very nice to have from a CX perspective but won’t improve the solution enough to justify the cost to the timebox.
- Won’t Have something that everyone still has in the backlog, but understands that due to constraints is not a feasible ask.
By meeting with your team and stakeholders and quickly bucketing work into these groupings, you can immediately build consensus and even start to vote on functionality to get contrarian perspectives on things we might not be seeing around the corner.
Four Futures
The Four Futures framework says that all the stories we tell about the future ultimately fall into one of four distinct categories:
- Growth. The systems and ways of being we live inside continue to develop along their current trajectory.
- Collapse. Our current trajectory comes to a sudden halt. Our systems and ways of being can fall apart.
- Discipline. New forms of restraint and control are imposed on the present order to prevent collapse.
- Transformation. Entirely new systems and ways of being are found; we transcend the present order.
Asking the team to come up with these worst-case and best-case scenarios, start to build around failure, and strive for the upside outcomes.
Growth. The trajectory you’re on right now continues upwards. This scenario is about climbing the ladder that’s currently ahead of you. What are the key landmarks along that journey? You get that promotion or new qualification? If you’ve recently been let go, you’re rehired in a great position in the same field? Move to a nicer apartment? Build your startup? What do you need to do to make that happen? Map it the most realistic sounding Growth scenario you can.
Collapse. Obviously, this is the scenario you want to avoid. But what does sudden collapse in your current trajectory look like? What are the forces — both inside you and in the world around you — that could cause that collapse? Does this economic crisis put your job at risk? Are new technologies putting your entire industry at risk? Are you working in a way that is unsustainable longterm? Are you carrying destructive emotions that will eventually catch up with you? No one wants to spend time thinking about personal failure. But map a realistic Collapse scenario, safe in the knowledge that this exercise is helping make it doesn’t happen.
Discipline. So you’ve identified the inner and outer forces most likely to cause Collapse. What would it look like if you acted to moderate or contain those forces? This is the Discipline scenario. It’s about changing ways of being and working, or seeking out new forms of support, to make your current trajectory sustainable for the longterm. How can you do that? What needs to be moderated or restrained? What new habits, ways of being, ways of working, or relationships could help in this?
Transformation. What would it look like if you transcended your current trajectory, and shifted towards something altogether new and transformatively better? This is the hardest scenario to map out, because by definition you can’t currently know everything about this new reality. But the seeds of it are inside you. Most likely, this scenario is about the passion project or creative career you (secretly?) dream about. Or the radical lifestyle shift — ‘let’s move to New Zealand and go off-grid!’ — that you sometimes consider or discuss with a partner. Map out the most viable journey possible towards that goal, and the most realistic picture possible of how it works when you get there.
Mapping out these four scenarios can be an enlightening exercise. Just thinking about your future in this structured way will offer you new insights into your current circumstances and the forces of change currently acting on your life. Most of all, the Four Futures provides a structured way of thinking about which future you want.
-David Mattin
1–2–4-ALL
This is an exercise where teams might have a lot of directional understanding of where they need to go, but they might not have a viable path forward with solutions.
This is included in a series of “Liberating Structures” exercises created by Keith McCandless and Henri Lipmanowicz. In short, the team is chartered with the problem statement and works independently to bring a solution. Then, in a timeboxed fashion, all are partnered with another peer. They take their two concepts, remix them, rinse and repeat with four peers, and then all the concepts are reviewed.
This builds on a concept I heard Julian Shapiro talk about on Lenny Rachitsky’s podcast, which he referred to as the “Creativity Faucet.” He noted that numerous talented musicians and writers force themselves to write down bad ideas, even when they see them as such, because it allows them to open up pathways to more creative ideas. Why? He theorized that after the bad ideas are emptied of your mind, your brain can instinctively start recognizing patterns of WHY they were terrible ideas and adjust accordingly.
This exercise increases the mind’s firepower, and teams can quickly start to see patterns and adjust solutions by building on top of one another without suffering from groupthink.
Google Design Sprint
And finally, my go-to framework.
The Design Sprint was created in the mid-aughts by a team of Google Ventures to quickly decide/test/iterate concepts in a five-day sprint.
It has been popularized and is likely one of the more familiar frameworks in Product Thinking.
“The sprint is a five-day process for answering critical business questions through design, prototyping, and testing ideas with customers. Developed at GV, it’s a “greatest hits” of business strategy, innovation, behavior science, design thinking, and more — packaged into a battle-tested process that any team can use.
Working together in a sprint, you can shortcut the endless-debate cycle and compress months of time into a single week. Instead of waiting to launch a minimal product to understand if an idea is any good, you’ll get clear data from a realistic prototype. The sprint gives you a superpower: You can fast-forward into the future to see your finished product and customer reactions, before making any expensive commitments.”
The phases as explained by the Google Design Sprint Kit:
Phase 1: Understand
In the Understand phase, you will create a shared knowledge base across all participants. Using the Lightning Talk method, knowledge experts across the business are invited to articulate the problem space from business, user, competitor, and technological angles.
Phase 2: Define
In the Define phase, the team evaluates everything they learned in the Understand phase to establish focus. This is done by defining specific context and desired outcomes of potential solutions. The phase concludes by choosing a specific focus for your Sprint, as well as goals, success metrics, and signals.
Phase 3: Sketch
In the Sketch phase, the Design Sprint team generates and shares a broad range of ideas as individuals. You will start by looking for inspiration, such as solutions in alternative spaces. Then, each Design Sprint participant will individually generate ideas for consideration. From there, the team will narrow down ideas as group to a single, well-articulated Solution Sketch per person.
Phase 4: Decide
In the Decide phase, the Design Sprint team finalizes the direction or concept to be prototyped. Each participant will share their Solution Sketch, and the team will find consensus on a single idea through decision-making exercises. The final direction will aim to address the Design Sprint focus.
Phase 5: Prototype
In the Prototype phase, the Design Sprint team will work together to create a prototype of your concept. This is when many decisions are made around what exactly the concept is and includes. You will aim to create a prototype that is just real enough to validate, and you will do it really fast!
Phase 6: Validate
In the Validate phase, the Design Sprint team will put your concept in front of users — this is your moment of truth! You will gather feedback from users who interact with your prototype, and if relevant, you will conduct stakeholder and technical feasibility reviews. You’ll end your Sprint with a validated concept– or an invalidated concept to improve on. Either way, you’ve made progress.
Putting it into Practice
So now that we’ve run through several frameworks let me take you through the one we here at Rightpoint like to execute: a Rapid Innovation Sprint or what we like to call a “RISE”.
Developed chiefly on client constraints, they often can’t set aside key stakeholders for five straight days. So we’ve developed a series of exercises that allow the team here to work asynchronously after 1–2 days of participation from the client. I’ve personally facilitated about 25–30 of these sessions during my time here, and these are my go-to exercises:
The Players
What we Know
I always start with a full recap of what we know. Unfortunately, we are often under tremendous time constraints, so my main goal is to bring the team up to speed as quickly as possible. It is comprised of three parts:
Challenge: clear definition of the problem to be solved.
Hypothesis: how our involvement could help mitigate the issue.
Opportunities: the tools/infrastructure/processes we have at hand to turn our hypothesis into action.
This can all be decently high-level and doesn’t have to be perfect. That is the enemy of good enough. This is a shortcut to get you on the same page as your client and get the directionally correct answer.
What is the Vision
The meat and potatoes are what this means to the stakeholders. I always ask for a future vision of what this solution will mean to the product two years later. Then we take all those answers, break them down, and boil them down into a vision statement. It’s a similar goal to 1–2–4-All, and we are writing down the North Star and editing it live as we see fit.
Obstacles/Doubts/Opportunities
Thinking of the future, how will things go wrong? I’ve also leveraged “Four Futures” to complete this in the phase, but also simply with stickies and columns with basic Obstacles/Doubts/Opportunities headers.
The right people in the room know where all the skeletons are in the closet, and as we start identifying patterns, we can determine what is a landmine and what is a goldmine.
Journey Mapping
Lastly, we take the main journey — it can be Back-of-House, User, or Service Journey and map it out. I love using miro and columns and then starting to string all activities together.
Amazingly, just writing out complex flows in front of clients starts to open your eyes when you see how many things have to happen to make one journey unforgettable. It also commonly identifies bottlenecks that might not have been able to be seen without an outside perspective.
The Goal
We will then start to heat map the journey and pick a goal that makes sense to tackle in the first sprint. That means we have all come together, met for four hours, and have a clear directive for a possible hypothesis that we can get started on iterating.
Build, Test and Measure
As soon as the journey is selected, the onus is on the product team to build, test and measure. Of course, results will rarely be perfect, and that’s ok. But the trust leaders have given to make something without a complete understanding of ROI means that the tradeoff is to iterate quickly to improve things and report results.
What are your best measurement tools? If your team sets itself up with a good foundation of analytics tagging, along with measurement and instrumentation, you can quickly get user feedback with the following metrics:
- Net Promoter Score (NPS)
- Customer Satisfaction (CSAT)
- Churn rate
- Retention rate
- Customer Lifetime Value (CLV)
- Customer Effort Score (CES)
I’ll leave it to my analytics experts here at Rightpoint, like Lee Rose to dive deeper into these concepts in the Rightpoint Mobile Apps Guide.
Bringing it all Home
So what is the correct answer for you? Like all things, it depends! A team that has an open mind and is willing to put themselves in sometimes painful areas might do better with intense Design Sprints. On the other hand, stakeholders who only focus on ROI would likely be better served with a MoSCoW Method.
No matter what activity you decide to execute, the one single goal is that you don’t need months of research to determine a direction and validate ROI.
And if you decide that you might need help executing any of these activities, please feel free to reach out and see how we might be able to run The Rapid Innovation Sprint as an accelerator for your next big idea.
Want help with your mobile app? The Rightpoint Digital Product team would love to learn more about the challenges your facing. Please reach out so we can set up a chat.
