De-Mystifying Product Analytics — An event by Advancing Women in Product (AWIP)
Credits: Advancing Women in Product Team — Nancy Wang, Karissa Barnett, Sarah Niehaus and Roshni Uppala
Product analytics has become instrumental in building a successful product that customers love. Along with our sponsored host of the evening Segment Analytics, Advancing Women in Product (AWIP) hosted a panel discussion featuring Sandhya Hegde (PM Director, Amplitude), Kate Zhang (PM, Facebook), and Lauren Reeder (PM, Segment) for a deep dive conversation on how to leverage product analytics to create successful products and user engagement. Here are some insights from the evening.
“Lead by example; be a data based decision maker!”
When setting metrics, start from the goals and ask the right questions
The most meaningful metrics are most likely specific to your particular product and company. For Instagram PMs, this might be: “How many repeat users did we have this month? How often did they post? What was the engagement received by other users?" These metrics are probably not relevant for a different product like Robinhood. Therefore, it’s very important to identify the goals of your product and feature first.
Then, think about what user behavior is a good indicator and predictor of the success of the goal, and what related metrics you can track for the user behavior. For instance, for YouTube, a key goal might be to enable users to find the videos they want to watch and the related metrics can be the amount of time users spend watching the videos. Note that when setting metrics, be very choosy and focus on the few metrics that really matter and can tie back to the product’s goal, organization’s goals and business results.
Test your product, early and constantly
It’s never too early to test your product ideas and there are different ways to do so.
- Leverage user testing and employee testing (a.k.a., dogfooding) for brand new product concepts.
- For complex decision making, qualitative data sometimes can bring more insights.
- Use A/B split testing for clarifying lightweight feature decisions. Some tips include:
- Make sure the two split groups are comparable and you have something specific to test, such as pricing, UI etc.
- Make sure the difference in results are meaningful and statistically significant.
- If the user base is small, make sure to give extremely different experiences for the two groups with multiple variable changes so that you can get an actual signal from the data.
- When analyzing the result, always break down the metrics to tease out the impacts from different components and find out the underlying drivers such as geography, time of year etc. For instance, when looking at conversion, break down the conversion funnel and see where the biggest gap is.
4. If you’re looking to measure more long-term impact, try hold-out testing instead. Different from A/B split testing, which splits the user base relatively evenly, hold out testing is an A/B experiment where you ship a feature to the majority of users and keep the rest from seeing the feature in order to measure the long-term impact. For instance, for a B2B SaaS company, hold out test can be a great way to decide whether to have the customer authenticate automatically via the UI or via the CLI. On the other hand, split test is more suitable to help figure out which UI layout delivers better user experience.
If you are interest in knowing more, take a look at here about hold-out testing and here about A/B testing.
If you’re on the hunt for the next Product Analytics tool, start with a free trial to show the impact
Of course, not every company has a data-driven decision making culture and not every company has the resource, infrastructure, or tools for data mining. If you are the PM for one of these companies, don’t feel stuck! You can start using the free version of an analytics product and just try it out. Do some analysis of a new feature and tell a story. Show the impact of analytics to different stakeholders in the company and help them understand how it can make their lives better.
Data will become more accessible, but the principles of product analytics won’t change
In terms of the future of product analytics, even with the exponential growth of the amount of data available and the increasing number of emerging analytics tools, qualitative data will never go away and will still be an indispensable part of any product decision making process.
With the increasing complexity of user interactions that these tools can capture, the questions product teams should ask to understand the user behavior and improve the product will change, but the fundamental concepts of product analytics will stay the same.
Analytics tools will eventually make data more accessible and everyone in the company will be equipped with the ability to both ask and answer questions directly with ease. It’s important for companies to provide collaborative learning tools and training resources to make sure everyone can take advantage of data to make decisions and unleash the power of data.
Interested in learning more? Then check these out:
- Analytics Academy by Segment
- Amplitude’s Blog Series on Product Analytics
Stay connected with AWIP:
- Sign up for our bi-weekly newsletter updates
- Join us for our next event “Building a Stellar Product Roadmap”
- Follow us on Facebook, LinkedIn and Twitter and stay connected with an amazing community of experts in Product Management!
