Blending qualitative and quantitative
Three Ways to Leverage Evidence In Product Management
Right brain, meet left brain
The role of Product Manager is a subtle one — crucial yet reliant on the power of persuasion, technical yet deeply human. As you try to tie all of the pieces together to get both global and granular about what you’re trying to build, influence becomes your true currency. Your effectiveness depends not only on finding the best path forward, but on your ability to get a whole range of decision-makers and team leaders to align with your game plan.
The good news is, getting buy-in doesn’t have to mean convincing your CEO. In fact, the right level of support from your VP of Product or team leads can prove just as potent, if not more so. But how can you rally that support to move your team towards a collective goal and vision? You can start by strategically using evidence to answer the core questions behind any new product management strategy.
- What problem are you trying to solve?
- Why does it matter to your existing or potential clients?
- How do you see yourself getting there?
Here are three ways you can leverage data to help you persuade your company to approve your product strategy and roadmap.
1. Hone your storytelling chops
The idea of building a narrative around a product or company is old news. But as our data collecting and analysis capabilities grow and mature, we’re learning to use evidence from product discovery activities to make increasingly accurate predictions and stitch together a whole new kind of data-driven story. Those stories may illustrate how users most active in the first 48 hours have the best chance of becoming power users. Or perhaps they walk you through a feature pain point that seems to be leading to a dramatic drop in usage for your first cohort of paid users.
No matter the direction in which you’re trying to steer the ship, evidence-based storytelling allows you to move beyond opinions and speed up your entire buy-in process by helping decision-makers absorb data, context and solutions through a narrative sequence they trust, recognize and understand. We could throw a whole myriad of statistics about the power of storytelling at you, but you’ve likely heard quite a few of them already. Messages delivered in story form, for example, can be up to 22% more memorable than basic facts.
It’s simple, really — our brain likes stories.
“Narrative is a key means through which people organize and make sense of reality and engage in reasoned argument (…) it is simply naive to believe that it is sufficient to present people with “the facts,” and assume that they will weigh these in a detached and rational manner and act accordingly. This view, the so-called “deficit-model” has indeed long been recognized as inadequate within the field of science communication.”
Brett Davidson, Open Society Foundations Public Health Program
So what makes for a compelling, convincing story?
As our brain best understands them, most stories feature a conflict, a resolution and a hero of sorts. With the right solution and visibility across all data silos, a great Product Manager should be able to identify data points to quantify and back up each element of that narrative. The conflict, for example, may be a bottleneck in your onboarding process, underwhelming conversion rates, or a worrisome crash report rate. Your solution might look like a better distribution of support resources, an unexpected but significant correlation between user behaviors, or a set of testing results demonstrating higher conversions with bigger buttons. And though the hero role might be temporarily played by a new feature or a streamlined process, data-driven storytelling also serves as a powerful reminder that ultimately, our end-users need to be at the center of any new product narrative.
“When we position our customer as the hero and ourselves as their guide, we will be recognized as a sought-after character to help them along their journey. In other words, your audience is Luke Skywalker. You get to be Yoda. It’s a small but powerful shift. This honors the journey and struggles of our audience, and it allows us to provide the product or service they need to succeed.”
Donal Miller, CEO, Storybrand
Another big challenge a good evidence-based story can help you tackle? Giving decision-makers the history and un-siloed visibility they need to make objective, informed decisions and understand your process, progress and reasoning.
“Each of us walks around with a bunch of stories in our heads about the way the world works. And whatever we confront, whatever facts are presented to us, whatever data we run into, we filter through these stories. And if the data agrees with our stories, we’ll let it in and if it doesn’t, we’ll reject it. So, if you’re trying to give people new information that they don’t have, they’ve got to have a story in their head that will let that data in”
Andy Goodman, Director, The Goodman Center
With the right data in hand, you can use evidence to thread the narrative of each assumption, experiment, failure, success, and iteration. With that story, you’re not only transferring knowledge but inviting new ideas that have yet to be tested. Evidence, after all, means not only understanding what works — but what doesn’t. Plus, failure data makes for healthy introspection and great stories: a daring idea boldly challenged, vanquished and then resiliently rebuilt and reinvented to tackle the problem once and for all.
Now that’s a plotline a great leadership team can buy into.
2. Balance your quantitative and qualitative data
Though we all tend to have a soft spot for hard, quantitative data we can easily measure and track, it’s important to also lean on qualitative evidence to answer a crucial set of softer, more nuanced questions that drive any great product. If quantitative evidence can help you answer the what and the how, qualitative information is your key to the why. Sure — your hard numbers can tell you that recently onboarded users are embracing a new feature far more than your early adopters, but it can’t tell you why. Were they attached to your previous UX? Was the feature more effectively explained in your initial onboarding sequence than in your broader introduction strategy? Is there another functionality they wish you’d prioritized instead? You need those answers to effectively redirect your team’s energy.
“Without knowing why we’re seeing a certain pattern in behavioral data, we might try to solve the wrong problem, or solve the right one incorrectly. Without validating gut feelings and intuition, we might go completely down the wrong path.”
Jeff Gothelf, Cofounder. Sense & Respond Press
Another way of looking at it, according to Gothelf, is to clearly differentiate between impact, value, and outcomes. While quantitative impact metrics like revenue and customer retention certainly sound sexy and can play a big role in getting high-level buy-in, he believes it’s the actual value created that yields the most powerful outcomes. Some of those value metrics may still be quantifiable — speed of task completion, number of functionalities used, session length — but qualitative evidence plays an important role in determining value creation too. Are users more likely to become ambassadors for your solution within their broader corporate ecosystem? Is your tool intuitive and easy to learn? Is your new functionality increasing brand awareness and industry credibility?
Quantitative behavioral metrics allow you to draw assumptions about what the answers to those questions might be, but with the right tool to centralize and analyze qualitative and quantitative data side by side, your softer evidence can come support or disprove those assumptions. As a result, your team can make more objective, data-driven decisions about your product and priorities. Beyond allowing you to move away from opinions and hunches, this careful balancing of evidence types also allows you to tighten up your continuous discovery loop; creating a structural framework that forces every member of your team to constantly revisit their assumptions and turn to the evidence to back-up their suggestions and arguments.
3. Build a product intelligence system to organize your evidence
While our newfound ability to aggregate and analyze thousands of data points might seem like a good excuse to hoard every bit of evidence you can find, it’s critical for a data-driven Product Manager to understand the difference between collecting data, and connecting it.
While some quantitative metrics may prove insightful on their own, most evidence proves to be much more instructive and actionable when contextualized, compared and correlated with data points from multiple sources and platforms. In fact, it’s at those intersections that you’ll often find the most valuable insights and opportunities for your product team — a pattern in users leveraging features in ways you hadn’t initially intended, or a surprise “desire path” between your product and another that might hint at the potential for a powerful partnership. But to uncover those motifs and overlaps, you first have to break down the walls between your data silos and build a framework that helps you determine causality and impact. Tools like GLIDR play an important role in helping you do just that; gathering product knowledge and evidence from multiple data sources including Jira, Trello and Intercom to give you full visibility over every moving piece of the puzzle. With that analytical power under your belt, it still falls upon a talented product manager to find the right balance between agility and structure in determining the best approach to data governance for their products and teams.
In a helpful illustration of what that data governance can look like, Amplitude Product Manager Dana Levine paints the portrait of two different approaches; one focusing on broad adoption through “information anarchy”, and the other emphasizing order and clarity through “data dictatorship”. Of course, it comes as little surprise that for most companies, the best strategy lies somewhere between the two.
“In the middle, we have a kind of data democracy. Typically when we talk about data democracy with Amplitude, we refer to everyone being able to access product data and use it in decision making. But this second type of data democracy involves enabling people to put data into the product analytics system. There are rules, but everyone is empowered to put in the data that they need to be able to understand their users and understand the results of their work.”
Being able to shape and customize your product management tool to give everyone the access, visibility and creativity they need to uncover new solutions and opportunities is key to fostering an evidence-based approach to product building while also leaving room for imagination and innovation. A flexible product intelligence system should broaden the range of data sources you can mine and explore. That flexibility, in turn, allows you to be as diligent about documenting your retrospective failures as you are about marking your successes, as nuanced in your understanding of qualitative data as you are in your interpretation of quantitative evidence, and as persuasive in weaving a compelling product narrative as you are in promising increased revenues and retention rates.
Start strategically leveraging your existing evidence to make better product decisions today. Try GLIDR for free and start validating or disproving your assumptions to help move your team towards a solid, data-backed vision of what you’re trying to build.
