Product Manager Interview: Analyze a Metric Change

Article originally published at Product Manager HQ on July 24, 2019.

One core responsibility of a product manager is to analyze the performance of her product.

After all, by analyzing product performance, product managers can drive impact by:

1. Determining how to further optimize traffic and conversion
2. Diagnosing bugs or problematic user experiences
3. Testing new strategies and new markets

During product manager interviews, hiring managers will focus on your analytics skill sets by asking you to analyze a metric on the fly.

I’ve had at least one of these metrics analysis questions for almost every single interview I’ve had! Your employer needs to know that you can handle the numbers, after all.

The standard product metrics analysis question usually looks like following:

“Conversion on our homepage dropped by 10% last week. How do you diagnose what happened?”

We recommend using the framework below to tackle this question.

1. Check the tool to make sure it’s accurate.
2. Check the underlying data to make sure it’s accurate.
3. Look back into history for any patterns in the data.
4. Consider recent changes you shipped.
5. Slice and dice the data.
6. Look for recent changes shipped by other teams.
7. Consider any changes that your company may have introduced outside of the product.
8. Look for changes in user behavior or customer trends.
9. Conduct competitive analysis.
10. Look for macroeconomic or geographical changes.

Why do we suggest this framework?

We suggest this framework because it’s mutually exclusive and collectively exhaustive, a.k.a. MECE. It covers all of the possibilities for why the metric could have changed.

On top of that, this framework is fantastic for highlighting your analytical rigor, your product strategy skills, and your ability to think in terms of systems.

The first principle of any metric is that metrics are only representations of reality.

In other words, metrics themselves can be flawed for many reasons: the tool could be broken, the data could be bad, or the metric definition may not make sense.

Once you’ve ruled out problems with the analytics themselves, then you can focus on the product itself.

Product management is all about working in concentric circles, so start where you have most control, which is the product that you’re managing.

If you rule out any problems with your product, start working outwards to other product teams, then to other stakeholders.

If you still can’t find anything within your organization, then it’s time to look outward.

Again, start where you have the most influence. Begin with customers and users, then look at competitors, then look at economics and geography.

The most important thing about using a framework is to tell your interviewer about your framework! Provide your framework at the very start of the question, before you actually talk through each component.

Give your interviewer your framework

Why should you tell your interviewer about your framework upfront?

Well, interviewers aren’t perfect. I’ve been an interviewer many times, and I’m happy to admit that I’m not a perfect interviewer. Given our imperfections, we sometimes ask questions that aren’t clear or questions that aren’t structured correctly.

So, if you can provide us your framework upfront, we can course-correct in the event that we asked a question poorly.

On top of that, we as interviewers want to help you succeed! If we knew you weren’t going to succeed, we wouldn’t invest the time into interviewing you.

Therefore, given that we want to help you succeed, we need to know how you’re thinking about the problem! That way, we can guide you toward success.

Say, for example, I asked you to diagnose why we saw a 15% increase in abandoned shopping carts for an ecommerce platform.

If you explained your framework to me, I might say “Let’s just focus on the data analytics — no need to conduct competitor research or customer research.”

From there, you can infer that you should be looking at seasonality, cohorts, pricing changes, etc.

Similarly, if you tell me the full framework but I really care more about competitive analysis, I’ll tell you to skip those other parts of the framework so that we can focus on just competitor actions.

So, now that we have a framework, let’s dive into each of the steps in the framework. Again, remember that you won’t necessarily tackle each one within the interview — especially because interviews are so short!

Check the tool to make sure it’s accurate

Is the analytics tool actually working correctly? Sometimes tools themselves have bugs.

True story — I once failed an interview because I assumed that the product had a problem, when it was actually the tool itself that was misreporting. As more and more product teams rely on third-party analytics tools, this class of analytics problems becomes more and more relevant.

If you do discover that the tool itself is problematic, ask about its past performance. If the analytics tool is regularly triggering false alarms, it’s worth digging deeper into understanding why it behaves the way that it does. That way, you can address the cause, or you can make a recommendation for a different analytics tool.

Check the underlying data to make sure it’s accurate

Once you’ve ruled out problems with the analytics tool, dive into the underlying data. Remember that analytics tools can only read the underlying data — and if the data is bad, then the metric is bad.

How is the data flowing into the analytics tool? Can you spot check individual data points? How did the development team implement the metric originally?

Be sure to systematically rule out any problems that might lie with the data itself! I’ve run into real-world scenarios where I assumed my product wasn’t performing well, when actually the problem was that I hadn’t implemented the metric appropriately.

Look back into history for any patterns in the data

If we know that the analytics tool is correct and that the data is correct, then we can now look into the data itself. But, before we consider the detailed performance of our product, let’s first look for patterns.

For the change in the metric — have we ever seen this pattern before? If so, that can provide lots of insight into why the metric changed.

Sometimes, the pattern is caused by seasonality.

Sometimes, it’s caused by deploying a new version to live production.

Sometimes, it’s caused by weather or by competitive actions.

Good product managers know how to pattern match. If you’ve seen the pattern before, you have concrete hypotheses as to what could have caused the change, and these hypotheses can then be quickly validated or disproved. Pattern matching saves you time!

But let’s say that you’ve never seen this pattern of behavior before. What now?

Consider recent changes you shipped

If both the analytics tool and the data are correct, and there are no historical patterns to explain the metric change, then it’s time to start analyzing whether your product caused the metric shift.

You have the most influence over your own product. So, did you recently ship a change? What was that change? Could that change have impacted the metric?

If you didn’t ship any changes recently, then your product probably isn’t the root cause. If you did ship changes recently, then it’s time to dive into the data to consider how your product caused the change.

Slice and dice the data

Good product managers know that data analytics isn’t just a science, it’s also an art.

While there are nearly infinite ways to cut the data, only a couple of cuts will actually matter. The tricky part: which cuts matter depends on the type of problem you’re solving and the type of product you own.

Here’s a real example.

One of my products was a landing page where visitors could submit inquiries and be connected to a real estate agent. Whenever we saw dramatic changes to conversion rate, one of the cuts we’d use is the zip code of the visitor.

That’s because real estate is highly geographical, and marketing campaigns are usually highly focused on a particular geography.

That’s not a good cut to use if you’re running an ecommerce platform.

For example, if you owned the landing page for Amazon Alexa, it’s probably not valuable to cut by the visitor’s zip code. It’s highly unlikely that any single geography would have changed in terms of conversion rate, because the Alexa product is relatively geography-agnostic.

Even if you did see a variance by zip code for your Amazon Alexa landing page, you’d assume noisiness — in other words, any correlation that you saw would likely be due to sheer chance.

The hardest part about analytics is mastering the art of analytics, not just the science of analytics.

By cutting the data in the right way, you can isolate the factor that actually drove the change.

But what if you still don’t see anything within your product that would point to such a drastic change in the metrics?

Look for recent changes shipped by other teams

Remember that your product is only one part of a much broader portfolio of products that your company owns.

Therefore, if your product performance changed even though you didn’t make any changes, it’s time to look at the rest of the portfolio.

What did other teams ship recently? Could their changes have impacted you?

Remember that your end user doesn’t know anything about your company’s organization structure, and doesn’t know which PMs own which features.

They just use the entire product portfolio. Therefore, changes in other people’s products can impact your product’s performance.

Look for business changes

Another principle to keep in mind is that product changes aren’t the only way for product metrics to change.

Any sort of business change could wind up impacting the metrics as well, even if nothing has changed about the product.

For example, did the business implement a new marketing strategy, with different messaging from the previous set? Did the business implement a new pricing model or a new sales strategy?

Did the business change the way it conducts customer support or operations? Any of these factors could lead to an impact on your product’s metrics, because business changes will wind up changing the behavior of the user.

But say that the business also didn’t change. What should you look at next?

Look for changes in user behavior

So, you’ve ruled out changes in your product and you’ve ruled out changes in the broader product portfolio. You also know that there weren’t any business changes that could have led to changed behavior.

It’s now time to consider external forces.

Again, start with where you have the most control. Begin with your end users and your customers. Conduct qualitative user research with them to see whether something has changed in terms of behavior.

Do they have different needs now? Did something change in terms of their pain points or their behaviors?

Conduct competitive analysis

If customer behavior didn’t change, then it’s time to consider your competitors.

Conduct competitive analysis on the industry. Did a competitor just join? Did a competitor just leave? Has there been a pricing change?

Look for macroeconomic or natural changes

So, your customers still have the same pain points, and your competitors haven’t taken any meaningfully different actions.

You can now zoom all the way out.

Consider the overall economy. Is there a recession, or is there a strong economy? Did politics or regulation wind up changing the landscape?

For example, consider a trade war. Trade wars can dramatically affect the impacted sectors. Say, for example, your product was in construction. A trade war that impacts the price of wood and metal could slow down the entire construction industry.

If it’s not politics, regulation, or economics, then the final factor to consider is nature.

Was there a natural disaster, or a sudden boon such as the end of a drought? These can all change customer behavior, which then winds up impacting your metrics.

Final Thoughts

Remember that this analytics challenge is meant to determine how structured you are when it comes to analytics. Therefore, you shouldn’t expect to make it through every point in this framework.

When practicing, practice under the same time constraints that you’d have in an interview.

Take no longer than 30 minutes to answer the question. Check to see whether you can explain your framework concisely, then dive into one or two of the areas of the framework — similar to how an interviewer would ask you to focus.

Another thing to keep in mind — product managers should be biased towards action.

Once you identify the root cause of a change in the metric (whether positive or negative), you should follow up immediately with action items. Find ways to exploit the positive upswing, or find ways to rectify a rapidly dropping metric.

Finally, remember to practice with other people! The PMHQ Slack community has a channel solely dedicated to scheduling practice interviews, so be sure to take advantage of this resource.

While practicing can seem daunting, remember that by practicing now, you will dramatically improve your performance during the live interview. Keep it up!

Interested in learning how to dominate these types of product manager interview questions and land the product manager job? You might want to check out our popular course: One Week PM.