A Framework for Measuring Product Impact
What is Impact?
Showing progress is a critical aspect of surviving and thriving in the product management profession. How you show progress and your team’s success will change as your product and organization grow. As you move from a tight-knit group of several people pre-Series-A to dozens, then hundreds and eventually thousands of colleagues across the company, this growing throng eagerly awaits news of what you, the mysterious product manager, are preparing to release to the world. As the enterprise grows and people’s roles are increasingly specialized it’s on us, the product folks, to articulate more and more crisply our mission and progress.
Netflix’s Culture Code from 2009 offers a useful lens. In their thoughtful deck they describe how enterprises deal with growing complexity in their business, particularly as the stakes become higher and higher. Reed and Patty argue that most companies become stuck-in-the-mud monoliths, the “big company” cultures that people speak of as being mired in “red tape and bureaucracy”. It’s important to remember that all that red tape was introduced by intelligent, well-meaning managers who logically added workflows, steps, checks and balances to ensure quality and alignment across the company. The downside of the “monolithic” culture is a lack of innovation resulting not just from the blockers that extensive processes introduce but moreover from the brain drain that inevitably occurs. Great makers do not ask for permission.
The second option Netflix offers is the independent silo model. This can work, they argue, but only in certain scenarios where the silos are effectively different companies. If it’s okay for the right hand not to coordinate with the left hand (usually if the products of each are distributed fully independently), then silos are a good option.
Netflix chooses a third option, where teams of high quality people align meaningfully, and enjoy loose coupling that supports an impressive pace of innovation. The benefits of this approach are undeniable: teams move quickly, free of the shackles of committee-based decisions, while continuing to incontrovertibly push the company forward toward their strategic goals. Commonly companies that use this approach share high-level goals across functional teams, similar to Kennedy’s “Man on the Moon” audacious goal, to which everyone in the country was able to connect their work to the larger mission. Facebook did an amazing job of this by declaring “Mobile” as a company-wide priority above all others (even turning the web version of Facebook off internally, forcing everyone to use their mobile devices). Google recently bolstered their commercial cloud product line by requiring all internal teams to build their products using their own cloud tools that were available commercially.
The point I’d like to make using this lens is that if you cannot align your teams toward the company’s goals, you cannot afford to leave them loosely coupled.
The foundation of creative freedom is a demonstration that everyone is on the same bus, headed in the same direction.
Transparency breeds trust
This phrase has become mantra for us on the HubSpot Product Team (hat tip Bradford Coffey) and rightly so. As we maintain our fast pace of development at ever-increasing scale (and stakes) it is key for us to crisply tie our product work to what the organization has set out to accomplish.
The habit of demonstrating visible and quantifiable progress generates cross-functional awareness and alignment, through trust.
As your company grows this discipline and transparency will help you maintain a swift pace of innovation, since your colleagues across the organization will trust you are solving the right problems.
How Do You Show Impact?
There are four levels of demonstrating impact and value creation during the product development cycle. Over the course of your product lifecycle as a Product Manager I recommend moving from level 1 through to level 4, doing whatever it takes to get there.
Level 1: anecdotes
Level 2: usage and engagement
Level 3: enterprise value
Level 4: customer value
The arc through these fours levels takes the PM from acting on instinct to demonstrating that the product is not only generating business value but actually changing the customer’s life.
At the very beginning of the journey developing a new product, you should be entirely focused on getting your first few people to love it and depend on it. It’s not yet the time to obsess over metrics or to over-analyze; as the Beatles said, all you need is love. For this phase customer anecdotes are your proof points (and/or early paying customers, letters of intent, etc). Document these early wins and share them internally — be sure to celebrate and build trust and momentum across the product team and company.
The shift from Level 1 to Level 2 is simply instrumenting tracking, and developing basic discipline around measuring engagement. Once baseline awareness is established for ongoing usage, common improvements are to begin to look at activation (what percentage of people who sign up actually receive the value they were promised?), and retention (as days and weeks go by, how well do cohorts retain usage in an ongoing manner).
The shift from Level 2 to Level 3 focuses on tying usage and engagement data to business outcomes. Regression analysis may reveal a connection between ongoing usage and lifetime value (through improved customer retention). Therefore this is a simple shift where PMs are still focused on moving the same levers but have a slightly more mature view on why — instead of usage for usage’s sake, the PM views user engagement through the lens of business outcomes.
The shift from Level 3 to Level 4 represents a dramatic shift in mindset that reflects itself obviously in the day-to-day work of the team. Level 4 shifts the focus from the value your company is getting out of your product, to the value your end users are getting.
An Example
Let’s imagine we are the PM for a “meetings” app, a straightforward web application that allows an end user to send a link for their contacts to book (available) time on their calendar.
At first, we may simply be trying to get our first happy human to depend on our new product. Success this early on may be low-volume anecdotal, qualitative feedback. We’re looking for “wow I love this, I need this!”
Once we’ve gotten our early smiles, we need to validate that people are actually using the product. We’ll add analytics tracking, and record important usage events, tracking them daily, weekly and monthly. As we get the hang of the reporting rig, we’ll likely add an activation event (booked a meeting) and several weeks in begin to look at whether cohorts of new users are continuing to us the product. We may stay in this stage for a solid amount of time, certainly through the beginning stages of monetization.
As time goes by and the product matures, our focus on analytics continues but is cast in a slightly different light. We find the features and usage patterns that tie closest to long-term paid user retention, and we begin to obsess over those. We look at SaaS unit economics alongside usage tracking.
Then one day we decide to be major ballers and begin to obsess over actual customer value. We still use usage tracking but we now use it differently. The thought exercise of “what is actually the most valuable to our users?” brings us to realize we should obsess over how many meetings our customers are getting booked, the quality of those meetings, and all the internal factors therein, like the “bail rate” of people no showing up to meetings.
At Level 4 we have a heightened perspective on what will deliver deep engagement and customer value. We’ll find pockets of users who have worked around current issues to get amazing results, and we’ll also find pockets of users who look like they are using the product but do not report getting much value from it.
Conclusion
I hope this framework will help PMs see that the level to which they are focused on the impact their products have is 1) a choice along a spectrum and 2) something that changes over time as the product matures.
While there’s no right answer or perfect way to measure product success it can help to have some options and a perspective on what fits your current challenges.
Thanks to Magdalena Georgieva, Hannah Fleishman, Dharmesh Shah, Lars Österberg, Bradford Coffey, and Jeremy Crane for their eyes and thoughts on this post as it developed.
