Photo by Snapchat/YouTube

How would you measure the success of Snap Map?

A Product Metrics Question for Product Management Interviews

StellarPeers
Jul 3, 2017 · 7 min read

by Malena Mesarina, Co-founder, StellarPeers


StellarPeers is a community platform that helps professionals prepare for interviews. We think the best way to prepare, is to work through questions and practice mock interviews as much as possible. We meet weekly to discuss product management interview questions on product design, product launch, strategy, marketing, pricing, and others. Last week, we worked on a product metrics question.

What is this question about?

This product management interview question tests whether your process of defining metrics is sound and structured. Do you tie metrics to goals? Are the metrics you specified actionable or lead to key insights? Do you prioritize which metrics to implement?

What is the interviewer looking for?

The interviewer is evaluating you on the following:

  • Did you begin by stating what the product goal is before talking about metrics? Talking about metrics without knowing the product goal is a red flag.
  • Is your answer structured or do you ramble?
  • Do you provide precise metric definitions or do you tend to give a general description such as, “I would find out if people used the feature frequently.”

How to structure your answer?

We suggest structuring your answer in the following way:

  • Ask clarifying questions to confirm your assumptions (e.g., What does the product do? Who is the product for?).
  • State or verify with the interviewer what the goal of the feature is (e.g., to increase conversion, revenue, profits, sales, engagement, retention, or other?).
  • Apply the product’s user experience flow to drive your analysis of what to measure. For example, think of what to measure when the user opens the app or opens a feature and interacts with it.
  • At each stage, identify behaviours that can be quantified with a metric(s).
  • Provide criteria to prioritize the metrics.
  • Summarize your analysis.

Answer Example

INTERVIEWEE: Okay, first I want to confirm that my understanding of Snap Map is correct. Snap Map is a location-based feature that is part of Snapchat, is it not?

INTERVIEWER: Correct.

INTERVIEWEE: And it enables users to share their location and Stories associated with those places with friends.

INTERVIEWER: Yes.

INTERVIEWEE: I think it works in the following way: a user opens the Snap Map feature by pinching on the Snapchat camera screen. A scrollable map appears displaying Bitmojis of friends around the world. The user can tap a friend’s Bitmoji to send a direct message or view any Stories the friend may have shared. Alternatively, the user can create a Story too.

INTERVIEWER: Yes, that is correct. And, in addition to friends’ stories, the map can also display stories featured by Snap. For example, news or events happening real time, like BART being on strike in San Francisco or a summer music festival nearby.

INTERVIEWEE: I see. So overall, it looks like this feature is a way to visualize Stories in a more lively way.

INTERVIEWER: Right.

INTERVIEWEE: In light of the increased competition from Instagram and Messenger Story features, this seems to be a step in the right direction. My guess is that the goal of the Snap Map feature is to increase user engagement?

INTERVIEWER: Yes.

INTERVIEWEE: Got it. So what I would like to do next is to talk about the user experience flow. The flow will help me identify which user behaviors are indicative of user engagement, and think of metrics to quantify these behaviors. The metrics will help monitor different forms of user engagement and evaluate if Snap Map is successful in increasing user engagement. Do you agree with this approach?

INTERVIEWER: Sounds reasonable, go ahead.

INTERVIEWEE: The user experience flow for Snap Map follows this process:

(The interviewee draws the following diagram.)

User Experience flow

I will go through each stage of the flow and talk about behaviors that arise. I will score each behavior on whether it represents a strong indicator of user engagement from 1 to 5. The score will help determine which metric to prioritize for implementation.

Leaving Snap Map

So let’s start with the user opening Snapchat and then pinching to open the Snap Map feature. One thing that can happen is that the user leaves without trying the feature. Not trying the feature could mean that the user did not understand how to use the feature, or they were curious and opened it but were not interested in doing anything more. Regular Snapchat users are savvy users, so if they opened Snap Map then immediately closed it is most likely because they were not interested.

If the user immediately closes the map, one reason could be that they didn’t see any Bitmojis on the Snap Map. To test if this is accurate, I would measure the following:

  • The fraction of times a user opened and left Snap Map per day and the average of this metric over all users. A decrease in this metric after launch would be a sign of higher engagement.
  • The average number of stories a user sees when they open the Snap Map feature. If the feature is successful in engaging users, this number should increase after launch and not be too small.

Knowing whether the user is not interacting with the feature is very important, so I would give these metrics a priority score of 5.

Tapping on Bitmoji

If the user doesn’t leave Snap Map after opening it, one thing they may do next is tap on a Bitmoji. I would measure this behavior by measuring the number of times a user taps a Bitmoji over the number of Snap Map sessions, and average them. An increasing average would indicate higher engagement. At this point, the user may or may not view a Story if there is one associated with the Bitmoji. Tapping on a Bitmoji is a behavior that shows a light level of engagement. We can call it the ‘curious’ level, and I would give it a score of 2.

Tapping on Bitmoji + view Story

Now say a user is more curious about a friend’s activity and taps on a Bitmoji. At that point the user can decide to view the friend’s Story or leave. If the user views a Story, that shows strong engagement, and it would be important to track that. So I would measure that behavior by calculating the fraction of the number of Stories viewed per person and also calculate an average over all users. I call this behavior ‘curious +’ and would prioritize these metrics with a score of 3, higher score than the previous metrics.

Ghost Mode

The user can prevent their location from being visible to anyone else on the map by turning Ghost Mode on. And if a user has Ghost Mode on all the time, that would be a strong indication that they probably don’t want to be tracked and don’t care about using Snap Map. I would measure this behavior by calculating the fraction of users that have Ghost Mode on per day, per week, and per month. If the Snap Map feature is successful in engaging users, this fraction should go down after the launch of Snap Map. This fraction metric is informative but is not a very strong indicator of user engagement. Because even if the user has Ghost Mode off, which means their location is being tracked, this does not mean that they are actively sharing Stories. They may have just forgotten to turn Ghost Mode on. So I would score this metric with low priority 1.

Creating Stories

The user can add a Story to Snap Map and share it with friends, which is an indication of a very pro-active user engagement behavior. I would measure this behavior by tracking the number of Stories created on Snap Map per user per day, and per month. If this metric increases after the launch of the feature, it would indicate that user engagement is increasing.

Users can also create Stories in the traditional way, outside Snap Map. And if the Snap Map feature is indeed successful then the sum of Stories created daily via Snap Map and outside Snap Map should be higher than before the launch of the feature.

I would give these metrics the highest priority score of 5 because they measure behavior that indicates strong user engagement.

Monthly Active Users

Market analysts like to use Monthly Active User (MAU) as a way to evaluate and compare competitive products, so I would define and measure this metric too. I would define MAU as the number of different users that did any of the following in a given month: tapped a Bitmoji, viewed a Story, or added a Story to Snap Map. Since this is a standard metric referenced by analysts, I would give this metric a score of 5.

Scoring of metrics

To summarize, the goal of the Snap Map feature is to increase user engagement. I proposed several metrics to track and monitor the achievement of the goal after launch. By analyzing the user experience flow, I identified user behaviors that point to user engagement. I also defined metrics to quantify those behaviors and prioritized these metrics based on their strength as indicators of user engagement. Based on my analysis, I recommend prioritizing those metrics related to these behaviors: leaving Snap Map without interactions, tapping on a Bitmoji + viewing a Story, and creating a Story. And as a metric to share with analysts, I would also keep track of MAU.

StellarPeers

To learn more about product management interview questions and to find practice partners, visit us at StellarPeers.com.

StellarPeers

StellarPeers is a community platform that helps professionals prepare for product management interviews.

StellarPeers

Written by

StellarPeers is a community platform that helps professionals prepare for product management interviews. https://stellarpeers.com

StellarPeers

StellarPeers is a community platform that helps professionals prepare for product management interviews.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade