Lessons Learned About What Questions to Use for Interviewing Product Managers
I recently tweeted this message about an interview question I’ve been asking Product Management candidates, off an on, for years:
The reason I tweeted it, is that I thought it was going to cause some discussion around what factors should be analyzed for something to count as a good answer. I wanted to hear some great analytical answers, some original ways to look at the question. What kind of clarifying questions people might ask? What I got instead was a full-on attack on why I would even be asking such a question:
My first instinct was to argue. I mean, this is Twitter we’re talking about after all! But then it got me thinking. What if there is something fundamentally wrong about Fermi estimation questions? Do they have a dark side? Every PM interviewer at Google is taught to ask these questions. Could such a great company have been so wrong for so long?
The thing is, you see, I’ve always prided myself on being prepared. I never go into a meeting, no matter how mundane, without spending time — at least 10% of the duration of the meeting, as a rule of thumb, more if the meeting is pivotal — writing down what I hope to discuss and answering what I might be asked. For job interviews, I pull out all the stops. I research the interviewer, and you’ll be surprised how well they respond to that. I research the company, the industry. I put myself in the position of the person about to interview me, and speculate what the questions might be. Then I answer them. I read up on publicly available questions. I spend time thinking about what I would do if I got the job, and what can I proactively propose.
And yes, I also practice with people. I even help other people practice at PMLesson. So that one hit a little close to home. It just never would have occurred to me that preparation has a dark side. I always figured that if I ask a candidate a question, and they give a great answer because they are well-practiced, then that should be a good thing because it would serve them well after they get the job when they actually come around to doing it.
Thinking about what Dare said, it made me realize how privileged I was. I got to work at great companies and build a great network. I saw people demonstrate the kind of behavior that is considered to be how a good PM should behave. I got the chance to see the gestures, the way people go up to the whiteboard, and the way they break down a problem.
And yes, there is some signal in how well people answer an interview question like that. You can learn a lot about how they think, whether there is a method in their approach to problems. You can pick up signals. But are they the right signals? do they mean this person will make a better PM? Because if they don’t then we should stop asking them. If all we end up doing is filtering for the person who practiced the most, then Dare is right — it is a Diversity and Inclusion problem, which I care about, deeply.
I draw a three important lessons from this episode:
- You should always run your roster of questions, even the ones you’ve asked many times and are calibrated on, by many different people. You might discover another point of view.
- You should also prefer questions about what people did then theoretical questions. “Tell me about a time you dealt with a business strategy problem” is better than the abstract version of the same.
- When testing PMs for the analytical hat, asking questions about how they would deal with a real problem (e.g. you are working at Youtube and watch hours are suddenly down, what do you do?) is preferable to the estimation type questions. I’m not saying I’ll never ask them again. Though I will definitely give them less weight, and so should you.
Life is full of surprises, even when you think you’ve been doing something too long to be surprised. The world changes when we leave room for what we do every day to change, even if we thought it was working fine. Maybe it’s time for us to move beyond Fermi.
