Data-Driven Product Management

manav chugh
Feb 1, 2021 · 6 min read

As a young Product Manager, it is important to be Data-Driven, but there are traps on the way. Being fooled by Data? Collecting too much Data, too often! Here I discuss, how to become Data-Driven but not drunk on Data.

I recently completed my Data Product Manager course offered by Udacity. Through my experience gained professionally and through these courses, one thing was clear from the start, if you can’t measure it, you can’t improve it. Data-Driven product management is core to the foundation of successful products being used by millions of users around the world.

In today's digital age, technology is more dominating than ever before. Users are always adopting new habits coupled with changing mindsets, data is integral to understanding the problem and delivering data-driven solutions. But on this journey, sometimes not being blessed with data scientists or UX researchers on the team, it can be hard to understand the data at hand. This can lead to risks like tracking the wrong metrics, building something which does not offer any value to the customer, and ultimately misusing or being manipulated by it.

Photo by Franki Chamaki on Unsplash

1. Measuring the right things

Product teams are on a ship in open water and metrics are the driving compass that helps them understand the desired outcome. At times, there will be a focus group of metrics for a particular area of the product. This might leave the other important areas of the products, which are not measured. Hence, it is important, to constantly evaluate that measured metrics line up with overall business goals. Otherwise, product teams risk going in the wrong direction.

2. Being patient and avoiding “vanity” metrics

We need to keep an eye out for “vanity” metrics. These are metrics that are being measured that make us look good, but do not correlate with overall success or tell anything about the customer experience. They are simply are not actionable.

Photo by Jon Tyson on Unsplash

Being patient is critical. Product improvements, often take time to have any visible effects. When we tie down a KPI to a product feature that is recently released as part of the release cycle, we shouldn’t start looking at it every hour. And when it doesn't meet the team's expectations, we shouldn’t change the KPI on the fly when it doesn’t immediately move in a positive direction. Humans are complex and behaviors are unique. Small things like weather, holidays, website outages, life in general, all affect a user's need to hire a product to solve a problem. So give time to users, to understand and adjust to the features.

If conversion drops on the first day after release on a new feature, adding more variables to it will not “fix” this. Be patient and give it some time. If not, all the effort and time spent will go to waste on seeing any measurable result that the team could have expected and you might be killing a change that would have made a positive impact later on.

3. Running Tests

The A/B test can be considered the most basic kind of randomized controlled experiment — Kaiser Fung

As part of these randomized experiments, we need to be sure that the sample size is not too small. This is because you need to achieve statistical significance. This will help you to verify the result that you are seeing, is not just background noise but also minimizes the act of this randomness through statistical confidence.

Have you ever run an A/B test and ended up with more questions than answers? More often than not, I have experienced the more variables, the more data you need for a statistically significant result. You need to be extremely precise on what you are testing and the associated impacts you can foresee.

Keeping it simple and testing one variable at a time, is the best way forward. If the test you want to run requires too many variables then you’re probably better off doing some qualitative testing first and putting aside any excess variables before setting up an A/B test.

In this process, it is important to measure the metrics that you have aimed for but also the other metrics which are going to be affected.

4. Qualitative and Quantitative Data

We want to create products that don’t only solve the user's pains but also that the user loves the product. Cognitive behavior and emotional connection play a deep role and are hard to measure.

That’s where qualitative research and data play a big role and provide product managers and researchers to understand the complexity that makes the emotional connection.

5. Product Analytics tools

Photo by ETA+ on Unsplash

But at the end of the day, product analytics is not about the tool but is about you. At times, we see that there is only one team “tracking” the metrics and no one else has access or visibility to these metrics.

In this way, it creates a dependency on one team and fills its backlog. This is the perfect recipe for resentment and a poor understanding of business issues. Instead of one dedicated team, ownership should be given to each feature team who are responsible for their own tracking, in autonomy. The shift in ownership of tracking the metrics for different teams leads to a similar shift in culture. On the other hand, it allows different team members to learn these tools and also have a go at the data being collected and ask questions.

These were just some thoughts that I wanted to write down. Of course, there are still many things about data and its pitfalls to be understood. But the idea is, to use data to track and validate the hypothesis but not expect that it is some “secret sauce” that will immediately improve all aspects of your product and that it will answer every question.

Understanding when to use data, and how much data to use, begins with first determining the importance of the decision at hand. Using data as the only strict answer can be pretty dangerous. But again. we are always learning. There is no perfect recipe for this but to understand quality is, even more, important than quantity.

Thanks for reading!

Dynatrace Engineering

Stories from the people who shape Engineering at Dynatrace.

Dynatrace Engineering

A publication by Dynatrace Engineering, sharing how we engineer, design & innovate, how creating value is what drives us and what the #DynatraceDNA is all about.

manav chugh

Written by

I am a product enthusiast working as a Product Manager. I have a brain of an engineer, the heart of a designer, and the speech of a diplomat.

Dynatrace Engineering

A publication by Dynatrace Engineering, sharing how we engineer, design & innovate, how creating value is what drives us and what the #DynatraceDNA is all about.