Using Data to build great products
Panel: Ellie Powers (PM, Chrome), Jennifer Liu (PM Director Shopping), Ali Miller (PM, YouTube), Shenaz Mistry (PM, Pixel) , Yvonne Chou (PM, Display Ads)
I had an opportunity to attend a panel talk along with women-in-product team. They brought together a diverse group of women who are Product managers of different type of products at Google. This talk was primarily focused on how product managers use data to make decisions. These are some of the highlights from the talk.
How to find important metrics to measure?
One of the approaches to take for projects is to frame OKR . OKR’s are a way to align the entire team to focus on the same North Star. First framework to think about metrics is to think of them in two categories think of hero metrics for your product and then critical metrics. Prioritization should be done to bring about long-term improvements in hero metrics. Another framework to think about metrics is to think about layered approach to metrics. On top there is Rally metrics : this is the overarching metric that can be used to evangelize team. Next layer is production metrics which are decided by connecting with the cross-functional team managing the project. The last layer is operational metrics, which are extensive log of user metrics.
How do you use the right data?
To choose the right data have clarity on the goal, which should highlight what are you trying to achieve. Once the goal is clear it is important to think of it through different perspectives such as that of a UX team member, engineer, customer. Another way to go about it is to build a hypothesis (quantitative or qualitative) and at the backend run experimentation to see incremental changes. For front end products it is important to look at user-flow before deciding on metrics. It is important to not just look at the quantitative data in isolation but to look at it in conjunction with qualitative feedback collected from the users.
How to not get distracted by over using data?
In order to prevent getting inundated by data clarity on what success looks like for the team is important. Once success for product is defined next it is essential to define how to measure it. For goals it is important to build a roadmap and tackle choosing right data over period of time then making hasty decisions and justifying using all data that is logged.
When is experimentation great to make decision?
Launch and iterate is the key to bring about incremental improvements to a product. This model does not work for physical products. A/B test is a good approach to make objective use of data. There are cases when A/B tests do not provide statistically significant data. In those cases it is helpful to couple the experimentation quantitative data along with user studies, user research data or to go with your gut to make decision. Experimentation is a great way to fail often, fast and cheaply.
How to utilize data to make decisions?
It is important to not assume that user is just like you. UX research is critical to leverage and understand diversity in user types. How data is utilized also depends on if you are leading the market or making the market. The limitation of data is that it helps give you a pulse on current market scenario but does not completely prepare you for user needs in the future. For understanding future needs talking to customers/users is essential. Implicit challenge with utilizing data is getting agreement from all the stakeholders. The way to face this challenge is to hear different perspectives before making decision.
How to influence stakeholders?
Transparency goes a long way to win trust of stakeholders.Having data along with user journeys help in framing convincing story for all the stakeholders. For stakeholder management it is important to understand what matters to them and present the findings accordingly. People management is as important as product management to be an effective product manager.
