Your interviews shouldn’t be spoilable

We ask a lot of a job interview –– given a few short conversations with a person, we attempt to predict what that person can contribute to the company over the course of several years. Every choice we make when designing the interview process should be focused on making that prediction more accurate. What I’d like to suggest is that in order to get the best signal, you should err on the side of transparency when preparing candidates for the interview.

Generally speaking, hiring managers give candidates little or no information about the interviews they’ll have. Sometimes the person being interviewed will know the names of their interviewers, but when it comes to the topics that will be discussed, they’re usually left in the dark. This has in turn create a cottage industry that exists to help people prepare for coding interviews and an anxiety-inducing process.

How much should you share with the candidate? As much as an ethical person would share with a friend who they referred for the job. I would fully expect someone to tell their friend, “They’re going to give you a hands-on-keyboard interview focused on identifying and fixing bugs in a React application.” I wouldn’t expect them to share the source code for the application used in the interview.

In practical terms, my recommendation is to provide a one sentence description of each of the interviews, so that that they can prepare for the interview in a focused way. Some examples include:

• “Technical (not hands-on-keyboard) interview in which you talk through the architectural tradeoffs of a large system you’ve worked on.”
• “Behavioral interview that will focus on your communication style and how you approach technical decisions.”
• “Hands-on-keyboard SQL exercise and a discussion of SQL optimization.”

The main benefit of this kind of preview is that enables the candidates to solidify their thinking on the topics to be discussed and refresh their memories before the interview. I’ve sat in plenty of interview wrap-up meetings where candidates are criticized for not providing specific examples or telling the same story to every interviewer. Giving them a chance to think about what they’ll talk about ahead of time should enable them to present a fuller picture of their experience.

Maybe you’re asking the wrong questions?

At this point, you may be saying to yourself, “A one sentence summary of the interviews we give would spoil them.” This is where I make the claim that if your interviews can be spoiled easily, they’re probably bad interviews.

If an interview can be spoiled, it means that the answers can be memorized. Questions that can be answered by reciting a set of memorized facts are bad interview questions. Your interview questions should require people to work through problems or to describe and analyze their personal experiences. In an interview, I want to see someone weigh tradeoffs, think through problems, or explain the lessons they’ve learned through experience. I don’t want someone to explain to me how the Java garbage collector works — I want to hear about how you identified an increase in garbage collector hangs as the reason for increased timeouts after a particular change. If you’re a manager, I want to hear about how you improved the diversity of your team, not a survey of the relevant blog posts on reducing bias in interviewing.

Furthermore, your spoilable interview is going to get spoiled. Some referred candidates are going to get the answers from their friends. People are going to post your questions on Glassdoor. People often bring their interview questions with them from job to job, just because you’re seeing a question for the first time doesn’t mean it’s not common across the industry. You may think all of your candidates start on the same footing, but sharing your interview details is the only way to be sure. The book Cracking the Coding Interview covers 189 programming questions. Are you sure yours isn’t on the list? Leetcode has 1600 practice questions available. Is the person you are interviewing a great programmer, or have they just seen the solution to the problem you’ve presented?

Finally, if foreknowledge doesn’t reduce the quality of your interviews, it means that you have to change your questions less often, which makes things a lot easier for you and your team. Not only will you spend less time coming up with new questions, but your interview panel will also get better at giving your standard interviews. Finally, it enables you to calibrate the interview results over a much larger set of candidates, better refining your model.

What are we really testing?

Maybe you think that interviews should be difficult, and that the element of surprise adds to the level of difficulty. I would argue that difficulty isn’t the right metric for interviews; the goal of an interview is to elicit a spectrum of responses so that you can assess the answers based on a rubric of some kind.

If we accept that interviews are an attempt to create model of a candidate’s future work performance, then we should also agree that the experience of interviewing should closely parallel the work the candidate will be doing. At work, we generally have time to prepare. Interviews are perhaps most like important meetings. When you’re invited to an important meeting, you usually know in advance what you’ll be expected to discuss, and will have a arrive with a pretty solid idea of what you’re going to say.

So long as your questions are highly relevant to the role and elicit a wide variety of responses, they’ll work. You’ll need to have a rubric (formal or informal) for evaluating those answers so that you can actually compare the candidates. But with the right questions and a solid rubric, question difficulty is irrelevant as a consideration. A question like, “Tell me about a time one of your ideas really didn’t work out even though you thought it would” is neither difficult nor easy, but it almost always yields a useful response.

In closing

Ultimately, interviewing is a data collection process, and it’s incredibly important to give yourself the chance to collect the highest quality data possible. For that you need interviewers who understand what data they’re supposed to collect and who are prepared to do so, interviewees who aren’t too anxious to present themselves at their best, and a set of questions that provide a fair assessment of the candidates. I’d argue that winnowing out the questions that you can learn the answers to ahead of time and being more transparent in what you share with interviews to help them prepare will make your interview process better.