How I prepared for Facebook PM interviews: Execution & Metrics
👉🏽 You can find my other notes on PM interviews here: Product Sense and Leadership & Drive.
Product Manager interviews test for 3 core skills — product sense, execution and behavioral skills. The format is different for different companies but the interviews test for the same core skills. The hard truth is that most interviews require you to think on your feet and respond quickly. Frameworks help with that. I’ve developed the following frameworks after doing ~100 mocks, giving several interviews, and interviewing several candidates.
Things to keep in mind while using these frameworks
- Be flexible: Most interview conversations don’t go as planned. Don’t force a framework.
- Practice! Practice! Practice!: Practice mock interviews with these frameworks, and mould them into what works best for you. Your ideal is not the same as someone else’s.
- Collect your thoughts: As a rule of thumb take a 20 second pause right after the prompt is given to you. It will help you collect your thoughts and decide on which framework to use and how.
Execution
This interview tests your ability to prioritize, identify the correct tradeoffs and set your team up for success. As a PM you need to create logical frameworks to make decisions against. There are 3 types of questions that are asked in execution interviews:
- Trade off between A and B
- Set goal for a product
- Root cause analysis
Common gotchas
- Incorrectly defining the time period of the metric (e.g. daily vs monthly)
- Picking ratios as top line metrics
- Not thinking about the product wide impact
- Not thinking about all the stakeholders involved
- Not thinking about the short and longer term impact
- Not thinking about the correct population to experiment with
Learn how to design metrics that withstand the sands of time. Without a solid understanding of good and bad metrics none of the frameworks will help.
Trade off between A and B
Making tradeoffs and decisions are an everyday occurrence for PMs. They have to make the best possible decision with the information available and pave the path for getting access to that information.
e.g. How would you decide between showing more ads on the Facebook News Feed vs. showing a People You May Know recommendation widget? (from interviewsteps.com)
Biggest gotcha: not identifying the common longterm goal between all the options
Framework to guide the conversation
- Clarify: understand what is the tradeoff, the status quo experience, which regions are you planning to make the change in, what population will be exposed to the change and why.
- Product and company mission: state your understanding of the mission of the product and how that furthers the company wide mission.
- Goal: state the goal that the tradeoff is optimizing for. For e.g. For FB Newsfeed we should prioritize content that is relevant for the user as measured by user engagement on the feed and with the posts.
- Key users & stakeholders: who are the internal (such as cross functional teams) and external (such as advertisers and consumers) stakeholders that will be impacted by this decision.
- Pros and cons: assume you go all in with both the paths (one at a time), what will be the pros and cons of each.
- Experiment design: layout your (strong) hypothesis and the population (such as users with more than the mean number of friends) you plan to test with.
- Decision framework: create a framework with the top line ****and counter metrics. Think of metrics in terms of buckets (e.g. advertisers, users, performance). Learn more about defining good metrics here.
- Tradeoff table: Prioritize the key metrics and state your hypothesis of how those metrics would move for the population that you picked, and what would be your expected decision based on that
- Summarize
Set goal for a product
As a PM you have to define the key success metrics and true north star for the team to prioritize against and push for. While defining the metrics, you need to keep in mind that the metrics should be actionable, measurable and not game-able.
e.g. Define goals for Google Meets
Biggest gotcha: picking vanity metrics as the top line metrics.
Framework to guide the conversation
- Clarify: understand what the product is and what product cycle it is in.
- Product and company mission: state your understanding of the mission of the product and how that furthers the company wide mission.
- (Operational/Analytical) Goal: pick a goal based on what cycle the product is in to focus the discussion: growth, engagement/transactions, retention or monetization
- Key users & stakeholders: who are the internal (such as cross functional teams) and external (such as advertisers and consumers) stakeholders that are involved with this product.
- Key actions: that you think should be encouraged and also the ones that need to be discouraged
- Metrics: top line and counter metrics to measure the key actions that you prioritized in the previous step. Think of metrics in terms of buckets (e.g. advertisers, users, performance). Learn more about defining good metrics here.
- Trade offs: what are the ways in which the metrics that you have prioritized can be gamed. Most people struggle with this. Before prioritizing the metrics in the previous step think about the tradeoffs you are making.
- Summarize
If all went well and you were able to prioritize the metrics, the interviewer will follow up with a situation where some of the metrics moved positively and the others moved negatively, and you will have to either perform a quick root cause analysis or determine next steps.
Root cause analysis
For a lot of PMs understanding metric fluctuations is a weekly occurrence (myself included). We have to be on top it, come up with hypothesis and validate. So its fitting that root cause analysis is a key part of PM interviews, and one of the most commonly asked questions. Treat the interviewer as a partner who has access to all the information you need.
e.g. Reddit traffic went down 5% — how would you report this issue to the executive team? (from igotanoffer.com)
Usually there is no correct answer (root cause) for such prompts. It’s about the breadth of investigation in a ordered manner.
Biggest gotcha: asking for information and stating the hypothesis without the why.
Framework to guide the conversation
- Clarify: understand what the product is and what the exact metric is. For e.g. “active riders went down by 5%”, clarify the definition of active and the time period the metric is computed over.
- Segment: break down the problem to see if it affects a particular region, platform, device, user type (such as new or returning, users in a loyalty program etc) or category (such as only Uber XL rides).
- Contextual: gather context around the metric fluctuation. Could it be seasonality, weekend-weekday effect, dig into the funnel, check the associated metrics (such as for rider cancellations see pick up times)
- Internal hypothesis: investigate data quality (e.g. missing/duplicate logs), product changes (direct/in-direct), degrading product quality (e.g. content on medium), changes in marketing and awareness strategy (e.g. cancelling promotion, decreasing budget in certain channels, in-product awareness pushes).
- External hypothesis: investigate competitor behavior (new product launches, new competitors), change in user behavior, market situation and regulatory changes.
