Photo by Janis Oppliger on Unsplash

What metrics would you use for Airbnb’s Experiences feature?

A Product Metrics Question for Product Management Interviews

By Malena Mesarina, Co-founder, StellarPeers

StellarPeers is a community platform that helps professionals prepare for interviews. We think the best way to prepare, is to work through questions and practice mock interviews as much as possible. We meet weekly to discuss product management interview questions on product design, product launch, strategy, marketing, pricing, and others. Last week, we worked on a product metrics question.

What is this question about?

This product management interview question tests whether your process of defining metrics is sound and structured. Do you tie metrics to goals? Are the metrics you specified actionable or lead to key insights? Do you prioritize which metrics to implement?

What is the interviewer looking for?

The interviewer is evaluating you on the following:

• Did you begin by stating the product goal before talking about metrics? Talking about metrics without knowing the product goal is a red flag.
• Is your answer structured or do you ramble?
• Do you provide precise metric definitions or do you tend to give a general description such as, “I would find out if people used the feature frequently.”

How to structure your answer?

• Ask clarifying questions to confirm your assumptions (e.g., What does the product do? Who is the product for?)
• State or verify with the interviewer what the goal of the feature is (e.g., to increase conversion, revenue, profits, sales, engagement, retention, or other?).
• Use the user journey map to drive your analysis of what to measure. (e.g., Think of what to measure when the user opens the app or opens a feature and interacts with it.)
• At each stage, identify behaviours that can be quantified with a metric(s).
• Provide criteria to prioritize the metrics.
• Summarize your analysis.

Answer Example

INTERVIEWEE: Okay, first I want to confirm that my understanding of Airbnb Experiences is correct. Is Experiences a new feature that lets travelers book a wide range of activities as part of their travel plans?

INTERVIEWER: That’s right.

INTERVIEWEE: And, what is the overall goal of making this feature?

INTERVIEWER: Well, I would like you to tell me what you think the goal is?

INTERVIEWEE: Experiences is a new offering, which aligns with Airbnb’s mission to make travelers feel like they “belong anywhere.” So the goal of this feature seems to be to diversify Airbnb’s revenue stream with a complementary offer that appeals to its base. This additional revenue can come directly from people booking activities, or indirectly from people prolonging their stays especially if they book multi-day experiences.

INTERVIEWER: Yes, I agree.

INTERVIEWEE: The success of the Experiences feature in creating additional revenue depends on how well it appeals to two parties: the travelers that book Experiences activities and the providers of the activities. So, I would separate metrics into two sets. One set will measure how well the feature appeals to the demand side, the travelers. And, the other set will measure how well the feature appeals to the supply side, the providers. I would like to start with the demand.

INTERVIEWER: That sounds good. Let’s start with the demand side, and if there is time, we will go through the supply side too.

INTERVIEWEE: Ok. On the demand side, a basic metric to measure success is the number of bookings of activities. But, besides this general metric, I would be interested in evaluating this feature on how well it drives the user towards that conversion. This is important to course correct anything about the feature that is not working well. So, I would like to create a journey map of the user, identify which behaviours indicate success or failure towards conversion, and define metrics to measure them numerically. I will then prioritize the metrics based on two attributes: how core they are to evaluating success and how actionable they are. By actionable, I mean, if the metric provides insights to improve something about the product. After prioritizing the metrics, I will wrap up with my recommendation.

INTERVIEWER: That sounds good. Please proceed.

INTERVIEWEE: Okay, I have tried the Experiences feature, but may I try it again before I start describing the user journey?

INTERVIEWER: Sure.

(The interviewee tries the Airbnb Experiences feature on her smartphone.)

INTERVIEWEE: Okay, here is how I would describe the user journey in the case of the website:

• The user logs into the site and sees the Experiences tab at the top of the page under the Explore Airbnb headline.
• The user clicks on the Experiences tab. A search page loads and displays a location search box with filters to search for activities. The filters are for Dates, Guests, Types of activity and Categories of activity.
• The user enters a destination location first and uses the filter tabs to look for activities.
• The results are displayed as a list of activities, and the user clicks on one or more of interest. Alternatively, the user drops off the site if nothing of interest can be found.
• Once inside an activity page, the user is presented with dates when the experience is available for booking, a description, five-star rating system and feedback from users. The user can book the experience or drop off.
• Social icons are available for the user to share the experience with friends.

In this journey, the user goes through an awareness, engagement, and conversion stage. And, there are two stages for post-experience: retention and referral. At each stage, the user takes actions that contribute to the goal of increasing revenue through booking of activities. I will list those actions and define metrics that can measure that level of contribution.

Two high-level metrics that measure success are:

1. Monthly bookings of Experiences activities
2. Revenue from monthly bookings

Although these metrics are not actionable, they summarize key business results at a point in time.

Now, I will go through the user journey. Could I use the whiteboard to work on my analysis?

INTERVIEWER: Sure.

(The interviewee draws the user journey map, and writes down metrics to measure through each journey stage.)

Airbnb user journey map of the Experiences feature

I would track all of these metrics on a monthly basis to compare month-to-month progress. Per month is a good time frame since any changes we make to the product would take at least a month to make a difference.

Awareness

Let’s talk about the awareness stage. People who arrive at Airbnb’s landing page may or may not know about the Experiences feature. But, clicking on the Experiences tab is a sign of interest. The number of different users that click on the Experiences tab at this stage would be our top of the funnel, and should be counted (#1 metric). The larger the funnel, the greater the number of possible conversions.

Engagement

Moving on to engagement — when people land on the Experiences page, they can filter activities of interest. Clicking on an activity detail page is a sign of interest, and a step towards conversion.

At this stage, I would measure:

The percentage of users from the funnel that clicked on an activity detail page (#2 metric). If this is low, I would try to find out why. Were there too few options? Was the experience not interesting enough?

The total number of clicks per activity page (#3 metric). This metric will help rank the activity pages. Commonalities among the top activities can advise organizers on what appeals to users.

Percentage of drop-offs (#4 metric) from the page with the list of activities. If the percentage is high, I would try to find out why. There could be a few hypotheses at this stage. Maybe the user cannot find anything they like. Or, perhaps all activities of interest were full. Or, maybe the price of the activity is too high for the user. So, I would measure the following correlations to test these hypotheses:

• The correlation between the number of activities displayed and the number of drop-offs (#5 metric).
• The correlation between the number of activities which were full and the number of drop-offs (#6 metric).
• The correlation between the price of the activities the user viewed and the number of drop-offs (#7 metric).

Conversion

Now let’s talk about the conversion stage. At this stage, the user is inside a single activity page and ready to book. I would measure:

• The number of bookings per month (#8 metric).
• The percentage of users at the top of the funnel that booked an activity each month (#9 metric).
• The percentage of users from the engagement stage that booked an activity per month (#10 metric).
• The percentage of drop-offs from the single activity page(#11 metric). If this number is high, I would try to find out why users left at this moment.

Retention

If the Experiences feature is successful, we should see an increasing number of activity bookings when users book a room. Therefore, I would measure:

• The total number of activity bookings over the total number of room bookings per month (#12 metric).

If cancellations occur, it is essential to know why. I would track cancellations when they happen and send a survey to users. I would also count the number of cancellations per month (#13 metric).

Referral

A way to measure if the organizers of local activities are doing a good job is to measure the number of referrals users are sending to friends about an activity. Therefore, I would measure:

• The number of referrals per user (#14 metric),
• And, the number of referrals per activity (#15 metric) to help identify the low performing activities and try to help organizers improve their offering.

I have fifteen metrics, but not all of them are core to measuring success towards increasing revenue or actionable. So, I will prioritize the metrics based on these two attributes with the help of a chart.

(The interviewee draws a chart to map the metrics into four quadrants.)

Core vs. Actionable metrics chart showing Core metrics in quadrants I and II

The metrics that fall into quadrants I and II are the most important because they are core to measuring the success of the goal. So, I would prioritize them over other metrics. The second most important metrics are in quadrant III. These metrics are not critical to evaluating success, but they are highly actionable. They are useful in finding out what to improve in the feature. Metrics in quadrant IV are not core and are not actionable, therefore not useful.

Okay, I will now evaluate each metric based on how core or relevant they are and map them to quadrants. Could I take a moment to do this?

INTERVIEWER: Yes, please.

(Row by row the interviewee enters a metric, evaluates it across the Core and Actionable columns, then assigns them to a Quadrant.)

Metrics Prioritization Table to help map metrics to quadrants

The number of users that clicked on the Experiences tab (#1) is core because it provides the top-of-the-funnel number, which serves as a base reference for other funnel metrics. Funnel metrics are important because they can help predict future bookings. So, I will put this metric into quadrant I.

The percentage of people that click on an activity (#2) is also a core metric because it is a funnel metric. This metric is not actionable because it does not provide insights into how to improve the product. Therefore, this metric falls under quadrant II.

The number of clicks per activity (#3) does not directly help measure bookings or revenue; therefore it is not core. And, it does not provide insights into how to improve the feature. So, I will put this metric into quadrant IV.

The percentage of drop-offs from the page with the list of activities (#4) is also a core metric because it is a funnel metric. It is not actionable because it does not provide insights into how to improve the product. Therefore, this metric falls under quadrant II.

The correlation between the number of activities displayed and the number of drop-offs (#5) does not directly help measure or predict the number of bookings, so it is not core. This metric is highly actionable though because it points to the need to list more activities, so it falls under quadrant III.

The correlation between the number of activities which were full and the # of drop-offs (#6) does not directly help measure or predict the number of bookings, so it is not a core metric either. However, like #5, it is highly actionable because it reveals that there is a need for making sure that users have multiple alternatives for activities. So, this metric falls under quadrant III.

The correlation between the price of activities a user viewed and the # of drop-offs (#7) does not directly help measure or predict the number of bookings, so it is not core. However, like #5 and #6, it is highly actionable because it reveals that prices may be considered too high by the users. So, this metric falls under quadrant III.

The number of activity bookings per month (#8) directly helps measure bookings and revenue, so it is a core metric. It does not help with insights into how to improve the product; therefore this metric falls under quadrant II.

The percentage of users at the top-of-the-funnel that booked an activity (#9) is a core metric because it is a funnel metric. It does not provide insights into how to improve the product, so it is not actionable. So, this metric falls under quadrant II.

The percentage of users from the engagement stage that booked an activity per month (#10) does not directly help measure or predict the number of bookings of an activity, so it is not core. And, this metric does not provide insights into how to improve the feature. So, it falls under quadrant IV.

The percentage of drop-offs from a single activity page (#11) does not directly help measure or predict the number of bookings, so it is not core. And, it is not actionable because it does not provide insights into how to improve the product. So, it falls under quadrant IV.

The total number of activity bookings over the total number of room bookings per month (#12) is one of those metrics that a CEO would be most keen on knowing. This metric is core because if this fraction is closer to 1, it means that most Airbnb users are booking activities every time they book a room. But it is not actionable since it does not provide insights into how to improve the product. Therefore, it falls under quadrant II.

The number of cancellations (#13) helps in measuring potential revenue lost. I would say it is core because it is directly related to revenue. But, it is not actionable, because it does not provide insights into what is wrong. So, it falls under quadrant II.

The number of referrals per user or virality rate (#14) is a core metric because it can help predict revenue. We can predict average revenue using funnel metrics in the following way:

• # of users next month = (current month top-of-funnel users #1) X (% that booked activities #9) X (Virality rate #14)
• # of bookings next month = (# of users next month) X (% that booked activities #9)
• revenue next month = (# of bookings next month) X (average price / booking)

This virality rate metric (#14) does not provide insights into how to improve the feature, so it falls under quadrant II.

The number of referrals per activity (#15) is not a core metric because it does not tie directly to bookings or revenue. And, it does not provide insights into how to improve the feature. So, it falls under quadrant IV.

So in summary, the metrics I would prioritize that fall in quadrant I and II are #1, #2, #4, #8, #9, #12, #13, and #14.

As a product manager, I would track all of these metrics. However, if I were to present to the CEO, I would pick the ones that will give the CEO a bird’s-eye view of the business. These metrics are:

1. The # bookings of activities per month (#8), which can be translated into revenue using an average dollar value per booking; and,
2. The total # of activity bookings over the total # of room bookings per month (#12).

Would you like me to continue with my analysis on the supply side?

INTERVIEWER: No, thank you. I think this suffices.

StellarPeers

To learn more about product management interview questions and to find practice partners, visit us at StellarPeers.com.