Crack the Data Scientist Case Interview by an Ex-Google Data Scientist
Whether you are interviewing for a data scientist, machine learning engineer, product analysts role, you will often stumble upon a business case problem.
Top companies such as Facebook, Amazon, Netflix, and Google love to ask business case problems in their phone screenings and onsite rounds. Other companies such as Twitter, Stripe, and Uber will ask such problems as well.
The formula for acing the business case problem is hazy. Unlike SQL and Python/R coding problems where the solution is definitive, a business case problem is open-ended with no clear approach to a solution.
Based on my interview experiences as a data scientist, formerly at PayPal and Google, I wrote this comprehensive guide that will help you ace the business case problem. Here’s my bio on LinkedIn if you want to learn more about me.
I am also an interview coach @ datainterview.com, a platform with practice questions and coaching services for candidates aspiring to land their dream data jobs.
In this guide, I will provide details of the following:
- What is a data science case problem?
- How do you respond to a data science case problem?
- How does an interviewer evaluate your response?
- How do you prepare for data science case interviews?
- My personal anecdote in solving a case problem in Lyft’s phone screening.
- A practice case problem from LinkedIn.
- Where can you find more practice problems?
Now, let’s get started.
What is a data science case problem?
Quite simply, a data science case problem is a real-life project you would work on if you were a data scientist at the company you are interviewing.
From building a recommender system, segmenting users for marketing, and designing an AB test — the project usually takes one to three months to complete.
Often, the role, team and company you are interviewing for determine the type of business case problems you could expect. For instance, for the machine learning engineer (role) in the fraud team (team) at Stripe (company), you could expect a case problem that requires you to solve: “How would you create a machine learning system that detects fraudulent merchants?”
Here are three more examples:
- Machine Learning Scientist, Product Demand at Amazon
“How would you build a forecasting system that predicts product sales?”2. Data Scientist, Product at Facebook
“How would you design an AB test on Facebook’s News Feed algorithm?”3. Data Scientist, Marketing at Apple
“How would you find a meaningful segmentation of customers of Apple?”During technical interviews, you are given 45 minutes or less to comprehend, solve and explain your solution to a business case.
Given the time pressure, candidates often get lost in how to solve an open-ended business case, costing interview points in methodology and communication.
The key to solving a business case is to understand an effective framework that helps structure your response.
I will cover the framework next.
How do you effectively respond to a data science case problem?
Imagine that you are an interviewee for a data science role at Apple. The interviewer asks you, “How would you find meaningful segmentation among Apple’s customers?”
How would you approach this problem?
You might not know the solution to the question right away. But, that is okay as, most often, the interviewer is not expecting it. However, candidates often make the mistake of providing a solution first without understanding and framing the problem. In addition, they ramble without clear focus and structure to their response as they feel the pressure to fill the awkward silence with ideas.
Keep in mind that a business case problem does not have a right or wrong answer. Two seasoned data scientists who are asked the same question will offer different technical solutions. Rather than expecting one definitive answer, the interviewer is assessing the soundness, structure, and substance of your response, off-the-cuff.
Let’s do a deep dive into how to respond to a case problem, effectively. Here are five steps:
Step 1 — Clarify
A common mistake is providing a solution without understanding the problem. Ask questions to understand the business context and define the key objective. For instance, you could ask the following questions to the interviewer:
- “What is the business objective of the segmentation? Is the application for product research or marketing?”
- “Do you want me to focus on Apple customers across businesses or within a specific business such as Apple Pay or Apple Music?”
Step 2 — Constrain
The next step is to constrain the open-ended problem into a key objective that you can solve. Constraining the problem ensures that your response is focused and shows that you understand the problem.
- “Based on your clarifications, I will focus on segmenting users of Apple’s subscription services such as Apple Music, Apple TV+ and iCloud. The purpose is to help Apple’s marketing team create customer profiles that can help them perform target marketing.”
Step 3 — Plan
Now that you acquired an objective, plan a response, if needed, on paper. You can jot down key points to convey to ensure that your response is structured, sound, and substantial. Ask the interviewer a minute or two to gather your thoughts:
- “I’d like to take a minute to gather my thoughts on a solution if that’s okay with you.” The candidate takes a minute to jot down ideas on paper. “To address this problem, I would like to discuss data preparation, statistical method, and a business recommendation. Are you okay with this?”
Step 4 — Method
This step is the crux of your case solution. The interviewer will assess you based on the soundness of your approach and the clarity of your explanation. The case problem asks you to “segment” customers. This should hint that a clustering algorithm is useful. Walkthrough the procedure on how you would apply the method:
- “First, I need to prepare relevant data that is useful for the segmentation. I could use user profile, user behavior, and marketing engagement data.”
- “Next, I will apply feature engineering to extract useful signals. For instance, I could extract the # of active sessions per Apple Music in the past 30 days. I can repeat the feature engineering on other subscription services. Such signals can help create what types of services a user typically uses.”
- “With the features identified, I can apply a clustering algorithm such as the K-Means. I can use the Elbow technique to find the optimal number of clusters. Finally, I can apply and interpret descriptive statistics on each cluster to create user profiles.”
Step 5 — Conclude
The last step conveys the “So What?” of your analysis. How can the business use your technical solution to solve the problem? Here’s a sample response:
- “With the clusters identified, I can provide a description of each cluster type. For instance, if a cluster contains users with low-frequency active sessions across Apple services, Apple’s marketing team could send a targeted campaign to encourage users to engage on subscriptions.”
There you have it. To effectively respond to a business case problem, follow the five steps: clarify, constrain, plan, method, and conclude.
How does an interviewer evaluate your response?
When you look at the interviewer, it feels as though you are receiving a blank stare. You begin to worry whether you are responding the way that the interviewer expects. Then, after the interview, you receive an email with an approval to advance or a rejection without explanation on what the interviewer thought of your performance. Here’s a behind-the-scene on how an interview evaluates a candidate.
Evaluation Process
During an interview, you will often see or hear the interviewer taking notes on a laptop while you are answering. What are they doing? The interviewer is populating an evaluation rubric for the hiring committee. Usually, the evaluation rubric contains the following sections:
- Question — A list of the interview questions
- Summary — A summary of a candidate’s answer
- Assessment — An evaluation of a candidate’s answer
- Grading Rubric — A grading rubric with ratings based on the overall performance
- Hiring Decision — A recommendation on whether a candidate should be hired or not
Once the rubric is populated, the interviewer submits it to the hiring committee which oversees the final decision on hiring.
Assessment
Whether a question is a case on ML or AB testing, characteristics of an effective answer that an interviewer looks for contain:
- Structure — Does the candidate frame the response in a structured manner or ramble?
- Completeness — Does the candidate provide a solution end-to-end, or is it incomplete?
- Soundness — Does the candidate provide a methodology that is sensible?
- Clarity — Can the candidate explain in a way that a target audience can understand?
How do you prepare for a data science case interview?
You learned about what is a data science case problem, how to respond to the question, and how an interviewer evaluates a response. Now, I will share tips on how to prepare for a case interview.
1. Research
More often than not, candidates overlook researching a company and the products before an interview. They have a general sense of the company, but they do not know the specifics such as the company’s mission statement, business model, and such.
To ace the case interview, it is vital to understand key details. Here are details you should gather before your interview:
- What is the mission statement of the company?
- What is the company’s business model?
- What types of products and services does the company offer?
- Who are the main users? What are the monthly active users? How is the “active” defined?
- What is the company’s pivoting towards over the next few years?
- What is the main function of the team that is interviewing you?
- What are the UI and UX elements of its core product?
2. Self-Practice
Practicing for a case problem can be tricky. Unlike solving an SQL problem, there is no definitive solution you can aim towards. On top of that, the case problem is conducted verbally. So, how do you practice? I will cover tips that I personally used to prepare and ace Google data scientist interviews. Here are the steps:
- Find a case problem — On datainterview.com, you can find 40 case problems covering case problems on statistics, product sense, AB testing, and machine learning. The questions are based on the ones asked in top companies such as Facebook and Amazon.
- Ideate a solution — On a Word doc, a sheet of paper, or whiteboard, jot down ideas on how you would approach the problem. Use the framework that requires you to clarify, constrain, plan, method and conclude to respond effectively, as mentioned in the previous sections.
- Explain the solution out loud — Time yourself five minutes, and provide an explanation of your solution. Explaining a solution out loud might feel awkward at first, but it will help train your ability to explain off-the-cuff, which is the usual experience of a case interview.
- Self-assess — Evaluate yourself based on the structure, completeness, soundness, and clarity of your response. Identify strengths and weaknesses in your response. For areas to improve, think about strategies that can help you provide a more effective response.
3. Mock Interviews
The best way to practice for a case interview is to practice with others. Sometimes, you don’t know what you are doing right or wrong. Practicing with others can provide the feedback you need to improve. Also, you may not feel the same pressure, practicing alone, as you would from an actual interview. Mock interviews with a buddy or coach can provide a similar experience, and, over time, you will learn to alleviate interview anxiety and perform well.
In datainterview.com, you can leverage two features:
- Mock Interview — This is a 60-minute session with 40 minutes of interview questions and 20 minutes of solution + feedbacks. The session is conducted by an instructor who worked at top companies such as Google, and the format emulated on interviews on such companies. Use this to level up your interview technique.
- Slack Community — If you enroll in the monthly subscription course, you become part of the Slack group where you can connect with like-minded candidates preparing for data interviews at top companies. Use the group to find a mock interview buddy.
My Lyft Interview Experience
In 2019, I was on my first technical phone interview with Lyft. The interviewer asked me this question:
“How would you measure the satisfaction of drivers on Lyft?”
I started by explaining the business context on why Lyft would want to measure satisfaction in the first place.
“I can tell Lyft cares about its drivers because you want to keep the drivers content and encourage them to keep driving on the platform. If the satisfaction goes down, drivers churn; if drivers churn, prices go up given the lack of drivers available; if prices go up, riders are less likely to use the service. Ultimately, the feedback loop will hurt Lyft’s business model.”
This first step was vital in demonstrating to the interviewer that I understand the business model of Lyft and the contexts behind why a question was asked.
I then provided a solution:
“I would send out a survey that measures satisfaction among drivers. The survey can contain a set of questions with ratings that measure satisfaction. The questionnaire can be dispatched on a monthly or quarterly basis among a random sample of drivers. One limitation of this, of course, is that the responder bias may cause the satisfaction score to be skewed.”
After my response, the interviewer asked a curveball:
“What if you can’t send out a survey?”
Here’s how I responded:
“Given that a survey isn’t an option, we need to identify an observable behavior that serves as a proxy for ‘satisfaction.’ I suppose that the closest proxy would be churn. If a driver is unsatisfied with the platform, they will leave. We can track the churn behavior among drivers and perform segmentation to identify types of drivers that are more likely to churn or not. A simple way to do this is to build a logistic regression model with features such as driver profile, engagement, geo and such.”
Lastly, the interviewer asked the following:
“How would you use the satisfaction measure to help Lyft?”
I had a decent sense of how to respond to this follow-up. When I was a consultant for the Marine leadership at the Pentagon, I studied factors that could attribute to suicide behaviors among the marines. We conducted a statistical analysis to potentially shape policies that can discourage such behaviors. So, I responded:
“Intervention. Here’s what I mean. If there are a subgroup of drivers that are more likely to churn, which would be a loss business for Lyft, perhaps they could be incentivized to stay longer on the platform with bonus or phone calls from the Lyft’s support team that address concerns that the driver may have. This added attention could encourage satisfaction among drivers and get them to provide more rides.”
I had advanced the technical phone screen after my case response.
A Practice Case Problem from LinkedIn
Problem
Duration: 15 Minutes | Difficulty: Easy
An interviewer at LinkedIn asked:
Two email versions were tested in San Francisco (SF) and New York City (NYC).
The total number of emails sent was 100,000 with 80,000 of those being email A while the remaining 20,000 being email B. The click-through rate (CTR) of email A was 15% while that of email B was 30%.
However, in SF, the CTR of email A was 15% while that of email B was 12.5% while, in NYC, the CTR of email A was 15% while the CTR of email B was 41.7%.
What is your recommendation on whether to launch or not?
Tips & Hints
This section provides tips and hints to help you practice this case problem, effectively. The practice tips highlight how to practice this question, effectively. The solution hints provide clues on how to solve this problem if you get stuck. Skip reading the solution hints if you want to try this problem first on your own.
Practice Tips
- To practice this problem effectively, set a timer for 15 minutes and explain your solution out loud to yourself or a practice buddy.
- Make sure to use a whiteboard, sheet of paper, or word document to jot down your ideas. Jotting down your ideas will help structure your response.
- Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
- As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.
Solution Hint
- First, assess the problem. Recognize the statistical phenomenon that explains the shifts in the CTR trends when comparing versions and versions x cities.
Solution
[Interviewer] Two email versions were tested in San Francisco (SF) and New York City (NYC).
The total number of emails sent was 100,000 with 80,000 of those being email A while the remaining 20,000 being email B. The click-through rate (CTR) of email A was 15% while that of email B was 30%.
However, in SF, the CTR of email A was 15% while that of email B was 12.5% while, in NYC, the CTR of email A was 15% while the CTR of email B was 41.7%.
What is your recommendation on whether to launch or not?
[Candidate] Before I provide a recommendation on which email should be launched, I would like to make sense of the campaign results. I see that the sample sizes of email versions within each city are missing. Could I have more information on this?
[Interviewer] Good question. Let’s say that the sample sizes are the following. Among the 80,000 emails being version A, 60,000 of those are sent to SF while the remaining 20,000 are sent to NYC. Among 20,000 emails being version B, 12,000 of those are sent to SF while the remaining 8,000 are sent to NYC.
[Candidate] Great, thank you. I would like to organize the results in a summary table to help me diagnose the problem. Please, let me know if my summary table is valid based on the problem you stated.
Commentary: Note that the candidate summaries the number in a table format. Case rounds are often verbal. In this particular case, the interviewer posed a lengthy question filled with numbers such that crucial details could be forgotten. The candidate takes the time to organize the details in a summary table as this approach aids her comprehension and shows strength in her data intuition to the interviewer.
[Interviewer] You got it. Do you notice any phenomenon that explains why the emails perform differently when considering cities? *Address this question.
[Candidate] Well, I see that when the emails are combined across cities, email B with 30% CTR clearly performs better than email A with 15% CTR. However, the performance flips in SF as email B CTR is 12.5% while email A CTR is 15%. I am sure that the performance is due to random chance. If you were to re-run the campaign, the city-level result could become consistent with the combined-level result.
Commentary: The candidate missed an opportunity to identify Simpson’s Paradox. Simpson’s Paradox is a phenomenon when trends changed directions when groups are combined. The phenomenon often occurs in experiments when control and treatment groups are evaluated across sub-groups such as countries and devices. Given that the paradox is basic in statistics, the failure to identify this can cost a point in the interview.
[Interviewer] Do you see any potential issues with the sample sizes not being consistent across groups? *Address this question.
[Candidate] Not at all. Given that we evaluate the results using proportions, this allows us to compare the email performances even though the sample sizes among the groups vary.
Commentary: The candidate made a faulty assumption that the varying sample sizes across groups do not pose any issues. The candidate could have pointed out two potential issues with the experiment.
First of all, was the sample size per group calculated using the statistical power before testing the emails? If so, the sample sizes should have been the same in versions A and B. Given the mismatch, the experimenter could have skipped the crucial step in determining the sample sizes before running the experiment. As a consequence, the standard errors of the CTRs vary greatly, potentially leading to a false conclusion about which version performs better.
Secondly, if the sample sizes were determined using the statistical power, the ratio between the two A and B should have been, respectively, 1 to 1, not 4 to 1. Such a condition in AB testing is called the Sample Mismatch Ratio (SMR) when the actual ratio of the sample sizes is different from the expected ratio, causing bias in results. The underlying cause could be an instrumentation error when users are not correctly assigned to the groups randomly.
[Interviewer] Okay. Now, let’s suppose that the different samples sizes do not pose issues. What could be other reasons that cities reacted to the emails differently? *Address this question.
[Candidate] I can think of two factors. First, the content in email B was more appealing to users in NYC than those in SF. Let’s presume that the email recommended jobs to users. The job openings were more appealing to users in NYC than those in SF. The poor email result in SF might indicate that the jobs recommender system requires calibration.
Another reason could be the macroeconomic factor such as the recession. Perhaps, the recession led to lay-offs in an industry. Jobs in SF are generally tech while jobs in NYC are generally finance. The recession might have afflicted a particular industry such that users who are geographically close became unemployed and receptive to emails with job openings.
Commentary: The candidate receives points on data intuition and communication. She provided a thorough explanation of the potential factors that explain the variability in the email results across cities.
[Interviewer] What is your recommendation on whether to launch or not? *Address this question.
[Candidate] I recommend email B to launch since it had a higher conversion rate of 30%. Even though the results vary by city, the variability can be ignored given that the global average for email B is higher than A. The ratio should generalize if the campaign was rolled out to more markets.
Commentary: The candidate made several key mistakes in her launch decision.
First of all, she failed to mention potential issues with low statistical power and sample mismatch ratio. Unless those are addressed, the results are biased, requiring a re-run with corrections.
Secondly, she disregarded the statistical and practical significance of the results. Though the conversion rate in email B was higher than that in email A, were the results significant statistically and practically to justify launching?
Lastly, she ignored the possibility that emails could perform differently by geography. The candidate should have proposed to investigate the underlying factors that led to the discrepancy in email results by city and propose re-running a separate AB test per city.
Assessment
The candidate is assessed across three attributes: statistics, data intuition, and communication. Each attribute is rated based on the following scale:
Outstanding — Quick response with a sound solution.
Good — Minor mistakes, but converged toward a final solution that is sound.
Borderline — Required several hints before providing a sound solution.
Inadequate — Incorrect response.
Based on the rubric above, the candidate receives the following remarks:
Statistics — Inadequate
The candidate made several critical mistakes throughout the interview. (1) She failed to recognize Simpson’s Paradox which is basic in statistics, (2) assumed that sample sizes being noticeably different do not pose issues to the experimental results, and (3) disregarded validity checks and statistical/practical significance before providing a launch decision.
Data Intuition — Borderline
In the beginning of the interview, the candidate showed competency in understanding the problem by creating a summary table of the email results. However, she faltered in making sound decisions based on the results as she was overly optimistic about the performance of email B generalizing well once launched. She should have assessed the results with skepticism, requesting more clarification and proposing to investigate further before making a launch decision.
Communication — Good
The candidate receives a good remark given that she clearly understood the problem and expressed her thoughts clearly for the most part of the interview. However, she failed to receive an outstanding remark given that her justifications in some answers were shallow. For instance, she failed to provide a comprehensive explanation of her launch decision.
Where can you find more practice problems?
For more prep content, check out datainterview.com :)
There are advanced tips on how to prepare for data science interviews and land your dream job at top companies such as Facebook, Amazon, Apple, Netflix and Google.
The flagship product, the monthly subscription course, contains the following core features:
- Case in Point — 40 data science case problems and solutions
- AB Testing Course — 12+ lessons on AB testing
- Mock Interview Videos — 4x1-hour recordings of mock interviews based on technical screenings at top companies.
- Question Bank — A list of statistics and ML questions commonly asked in interviews.
- SQL Drills — SQL problems and solutions to help you ace SQL rounds.
- Slack Study Group — Network with a community of job candidates and data science instructors who work in FAANG companies.
