What “Data-Driven” Looks Like in Product
To succeed as a high-growth technology company, you have to be able to show progress. Everyone’s got a great idea with an enormous market opportunity — the question is whether you can execute. The startup scene has become like an episode of The Bachelor with entrepreneurs desperately jockeying for the admiration of a handful of investors and early adopters. Instead of a rose ceremony, you get money to keep your lights on a little longer. Resources are awarded to teams and projects that demonstrate continued success. As a product manager that doesn’t actually write the code, close the deals, or generate the leads, you need to prove that your decisions are making a difference.
It’s not new thinking that product should rely on data, but with all the OKRs, KPIs, OGSM, and other alphabet soup floating around, we felt it was worth exploring just exactly how teams use data to win in their markets.
Earlier this month, Boston Product, in conjunction with Toast Inc, hosted a panel with three veteran product managers — Jennifer Burner (HubSpot), Jon Gilman (Runkeeper), and Oliver Young (LastPass) — where they shared five basic principles and skills that help them define and measure critical metrics to make smart, informed decisions and drive their product management.
Choose the right metrics
Product management in its core is solving problems. However, without metrics, it’s difficult to identify what kind of impact you’re making in addressing those problems. A good metric is one that’s aligned with your business objectives and one that your team can actually affect. For example, a good metric would be retained users, because you can ship features that keep users coming back on a regular basis. A more difficult metric might be “percentage of users that access from the airport.” As Jon shared, you want to “know exactly what game you’re playing to determine how your scorecard looks.”
Jennifer is in charge of HubSpot’s website building tools. When it comes to selecting metrics, Jennifer divides metrics into two different categories. The first are her core metrics which revolve around the essential parts of her product that seldom change. The second are project-specific metrics that assess the health of individual efforts to move the bigger needles.
The core metrics span over the entire product and are assess the general quality of the product. For example, Jennifer’s product needs to be fast, easy to use, and built for marketers.
These are broad problems to address, so how do you go about measuring your progress against each of these themes? To measure how fast their product is, Jennifer monitors the time to value — the amount of time between creating a new landing page and when the user generates their first lead through the landing page. She keeps a pulse on the incoming support contact rate as a way to measure her product’s ease of use. An NPS survey is conducted between the user base to gauge how well-built the product is for marketers.
Rally the team around the problem (and the opportunity)
As the product manager, you own the problem, and it’s your job to rally your entire team around the problem. Data is your friend here. Where a problem may seem intangible, you can use data to illustrate the scope or severity.
Many PMs begin new projects with some sort of requirements documentation that cover the what and the why. Metrics belong in those documents as well, so the team can understand what success looks like. Don’t limit yourself to the metrics you already have. If you need to measure something new to assess a project’s progress, put that into the requirements. As Oliver reminded us, “If you don’t have a clear understanding of what you’re trying to move as a business, it’s very hard to get anything done, even on a particular roadmap.”
Know what NOT to measure
As your team works to address a problem, it’s equally important to define what your team is not going to do. It’s very easy for a team to get sidetracked by an external factor, and you can define scope by explicitly defining what doesn’t matter to a project. For example, let’s say you’re trying to improve adoption of a little-known feature among your new users. You might explicitly say that feature usage by people who have been users for over three months is not relevant at all. Later on, if you find that overall adoption of the feature is up, but only by longtime users, it might be tempting to declare success, but you’ll know that wasn’t your initial objective.
At HubSpot, they have something called the MSPOT framework. The O stands for omissions, and these are defined as what HubSpot will not be pursuing each quarter. If a specific feature was listed as an omission, even if data comes in where users are clamoring for that feature, Jennifer’s team puts that data to the side so they can focus on their vision for the quarter. That feedback is valuable, and informs the roadmap going forward, but it also means that Jennifer’s team won’t be derailed from pursuing their quarterly goals.
Data is rarely perfect. Many external factors can skew your data or cause it to become unreliable for any given length of time. As the product manager, you need to know whether this bad data should be tossed away or adjusted.
One day at Runkeeper, Jon checked his dashboard and saw that web signups had skyrocketed well-past the number of in-app signups. Jon knows that most of their user growth comes from their mobile app, so he decided to dig deeper into the data. He learned that the spike was caused by new users signing up in China. Odd, but not impossible. They kept digging and found that none of the new web signups had downloaded the mobile app and that all the email addresses used to create the user accounts belonged to the same domain.
They ended up deleting the bot-generated users and cleaned up their data so that it wouldn’t be skewed. Without their intuition, they could have had the risk of taking the data to leadership saying that China was the next big market for Runkeeper, which was not the case.
Distinguish between reversible and non-reversible decisions
Being a PM is all about decisions. You won’t always have all the information you need to make the call, and in those times, you need to recognize what kind of decision you’re making. Oliver shared a quote from Jeff Bezos’s 2016 investor letter, which he considers the smartest thing about product decision-making he’s ever read:
“Some decisions are consequential and irreversible… We can call these Type 1 decisions. But most decisions aren’t like that — they are changeable, reversible — they’re two-way doors. If you’ve made a suboptimal Type 2 decision, you don’t have to live with the consequences for that long.”
You can afford to be wrong about Type 2 decisions, because they’re reversible. An organization’s size, personality, and leadership can impact the kinds of decisions that tend to be made. Jeff’s quote continues:
“[T]here seems to be a tendency to use the heavy-weight Type 1 decision-making process on most decisions, including many Type 2 decisions. The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention.”
As a product manager, you want to make as many of your decisions Type 2 decisions as possible.
Get a metrics routine
Best practices and real-world examples are helpful, but metrics require discipline. Everyone on stage advocated that PMs incorporate metrics into their daily or weekly routines. Jon checks his dashboards with his morning coffee. Oliver asks his team members to update spreadsheets manually because it’s a forcing function to look at the numbers. Jennifer has a system for moving data from their integrated dashboards into spreadsheets to answer specific questions. Each of these routines affords the team the full benefits of being data-driven:
- Prevent problems and successes from going unnoticed
- Produce a constant stream of new ideas
- Develop data intuition and improve decision-making
- Align teams around what really matters
During the Q&A session, Jon offered, “It’s so easy to get lost in the day to day as a PM — What are we working on now? What’s being shipped? — that if you can create rules around when you look at metrics and what metrics you look at, it makes it much easier to have data be a part of your daily life and not just something you remember to do one off.”
It’s important to take a look at your metrics over different time intervals. You may not notice a subtle slowdown in engagement on a daily basis, but it will be very easy to spot on a monthly timeline.
In conclusion
If you’re thoughtful about which metrics you pay attention to and incorporate them into your team’s processes and routines, you can build a culture of being data-driven, and that will help you thrive in the market.
If you like what you’ve read, join our community to stay in touch and hear about future events.
