How to Build a Data-Driven Product Roadmap
A guide to impact-focused opportunity seeking, with real-life examples of how we do it at Simply (Formerly JoyTunes).
Just a few months after joining Simply I got the ownership of all the data analysis for my team’s quarterly planning. My team works on a feature within our popular piano learning app Simply Piano. The feature called Play offers a vast sheet music library with hundreds of songs adapted to different levels of proficiency. Our KPI is user retention, and my job was to figure out how we reach the ambitious goal of improving our usage retention by 100%.
In this article, I want to share with you how we built an impact-focused product roadmap to reach this very ambitious KPI. Spoiler alert — It has to do with leveraging our data.
A Well Defined KPI Can Work Wonders
Vast amounts of literature has been written on how to choose the right metrics for your product. While I won’t cover it in this article, it’s important to try and define one, clear KPI. Organizing our teams around a single KPI did miracles for us to focus the team and reduce complexity in prioritizations (You can read more about our structure in this post by our CEO).
That’s why the first thing we do is single out where we are heading with a well defined KPI.
After defining our KPIs, we set an aggressive goal, which is capped in time. Aiming high forces out of the box thinking and leaves out traditional, incremental improvements. It brings to the table creative ideas, makes us focus on what is impactful, and rules out small, nice-to-have efforts.
Find The Levers
Identifying the right KPI levers is where we, the data professionals, can shine. In other words, we are searching for what influences it the most and what it consists of. To do so, we apply every tool in our toolbox to break big problems into local metrics that we can measure better and connect them to concrete efforts.
Our toolbox can consist of data mining, extensive analysis and experimentation, logical deconstruction, and predictive analytics and they all serve the goal to understand which metrics influence the chosen KPI and which are not. This is crucial to help boil things down to an actionable plan.
The data helps us support or negate our hypotheses and suggest new ones as well. When uncertainty is high, as it often is, we will run experiments and test our hypotheses about the connection between the different local metrics and our high-level KPI to establish more accurate sub metrics.
How does it work in the real world? A case study
Anyone who’s had to make important business decisions can remember the overwhelming number of roads that are there not to be taken. I’m going to outline a process that resulted in a clear and ambitious product roadmap despite all the uncertainty involved.
I’ve applied the suggested framework to our KPI of Usage Retention, calculated as the % of learners who are still with us after X months.
Through analysis, we identified that the three main levers of retention are:
- Reach
- Activation rate — The % of learners that actually find value in the product after being exposed to it
- Engagement rate
After establishing that, we can accurately estimate how an improvement in each of these metrics will influence our high-level KPI.
Next, we got the whole team involved in raising ideas that they think will pull these levers, each bringing their domain’s perspective. This framework helped us focus, leave the irrelevant features for later, and have an objective discussion about the impact of every feature. According to our experience, knowledge, and context, we’ve picked the leading ideas for impact analysis by the following parameters:
- Metric — To which metric will it contribute? What is the weight of this metric in the high-level KPI? (Some features can contribute to more than one metric)
- Population — To what % of the population will it be relevant?
- Expected Improvement — the lift we predict the feature will have on the treatment population.
Identifying the relevant metric and population is pretty straightforward, it’s the expected improvement that is trickier. Here are some of the practices we apply to help us out:
- History — Use past, similar efforts in these areas as a baseline.
- Elimination — Estimate the higher and lower cap of influence of this feature. Many features, even in the most optimistic scenarios, won’t bring enough value.
- Confidence — Some features have high confidence of impact (e.g. translation of the content to foreign languages) and others don’t (e.g. introducing new pedagogical concepts). We add estimation confidence as a coefficient. It helps to go for directions that have a higher probability to impact.
Let’s analyze the impact of an actual idea that was raised in the team: Personalize our onboarding flow for users with prior experience.
M: Activation.
P: ~20% of our learners are people that played piano before, based on recent cohorts.
EI: 10% is the maximal impact on activation based on previous efforts of a similar size in this area.
Maximal Impact = 2%
Ultimately, we end up with a list of our leading ideas scored on the same scale, which helps us prioritize them. We can now compare efforts such as developing the product for a new platform (reach), onboarding flow improvement (activation), or adding more advanced learning tools (engagement) on the same scale and prioritize them accordingly.
At last, we’ve estimated the effort for each of the leading features, to understand their scope and prioritize them tactically. Keeping effort estimation to the end keeps us on the strategic level, without getting tempted into many small, low effort but also low impact tasks. What we learned from this process is that at this point in time there’s significantly more impact in focusing on reach than on activation
Summary
The main ingredients of a well-focused roadmap are:
- Definition of main KPI and set ambitious goals
- Identification of the main levers of the KPI
- Understanding how each of the potential efforts is pulling these levers
- Choose the efforts that make the most impact
Any product roadmap without supporting data is at risk of losing focus, prioritizing the wrong efforts, and wasting the team’s precious time. We, as data people, are here to maximize our team’s impact which in turn brings more value to our learners. Time is the most precious resource we have, and we should use the different tools and frameworks to make sure we get the most out of it. Naturally, when dealing with hard problems no process is perfect, which is why we remain humble in our assertions and have in place quarterly touchpoints to revise our assumptions and decisions.
Stay tuned for additional posts that will explore what the day-to-day of a data person at Simply looks like and check out our open data roles.
