Conducting Great Interviews, Part III— References, Debriefs, and Decisions
Good interviews are like puzzle pieces. Each interview shows a different aspect of the candidate. And like a puzzle, having the right pieces is necessary, but insufficient — you still need to know how to put them together. In other words, a great process, for translating feedback into decisions, is key to great interviewing.
This is the last in our 3 part series on Great Interviewing:
- Part I— Preparation
- Part II — In the Moment
- Part III- References, Debriefs, and Decisions
We believe that reference calls are an indispensable piece of the puzzle. But why do them at all? Critics argue any candidate with a brain will surely choose laudatory references. That’s true. But if you’re thoughtful about who you call and what you ask, deriving meaningful information is not just possible, but probable.
First, who to call: two or more (former) managers of the candidate. Not peers or subordinates; managers have much more insight into the strengths and weaknesses of their direct reports. Request at least two, to see if the feedback is aligned. If it’s not, that can be a red flag that you’re hearing platitudes. And check LinkedIn to make sure the reference was actually the candidate’s employer — one of our favorite recruiter bloggers writes, “Over the years I’ve caught many candidates who gave me the names of so-called previous ‘managers’ who turned out to be no more than colleagues or even just friends!”
Mutual connections who are previous colleagues of the candidate can be good sources of perspective. These unsolicited or “backdoor” references are controversial, and they’re often done poorly. When done well, however, they provide valuable information, and reduce the concern about platitudes coming from candidate-brokered references. Here’s how to do unsolicited references well:
- Don’t call anyone at the candidate’s current employer!
- Consider the context: how long ago and for how long did the reference work with the candidate? Was it in the same field? Don’t call someone they worked with 10 years ago or who worked with them in an unrelated field.
- Don’t ask about anything you wouldn’t ask about in an interview (race, religion, etc.)
- Obtain written permission from the candidate for all types of reference checking, early in the process.
- If you can’t find any mutual connections, don’t hold that against the candidate.
Second, what to ask: questions that focus on actual past performance, not hypotheticals. Don’t ask the reference if they think the candidate will succeed at your company — they don’t know, nor should they. Here are 2 good questions to ask:
- What were the candidate’s major responsibilities when they worked for you? See if this aligns with how the candidate described their past work.
- What was a development or improvement area for this candidate? If they dodge it or give a stock answer, reference a development area you’ve learned about directly when interviewing the candidate.
If at all possible, do your references before you debrief after the interview. Granted, this has downsides. You have to delay the debrief for at least a few days after the final round of interviews. And it leads to reference-checking some candidates who don’t get offers. (Don’t reference check candidates who were clearly bad in the on-site!) But it’s far outweighed by the improved decision-making that comes from having good reference call information present for the debrief. Conversely, if you do reference checks after the debrief, you risk confirmation bias: looking for evidence to support the offer you’ve decided you want to make, and ignoring evidence to the contrary.
The debrief
At this point, your interviews are done and so are your references. You have all the information. But how the hiring team synthesizes that information is the most important part!
It’s best to bring together all interviewers and someone who wasn’t an interviewer: a moderator. The moderator could be a “bar raiser”, a practice that originated at Microsoft, and became famous at Amazon. There, the bar raiser is an experienced employee in the general area of the requisition who is not the hiring manager. In addition to running the debrief, the bar-raiser has veto power over the candidate — veto power not even the hiring manager can overrule. The veto isn’t a fit for all organizations, but the general principle holds: an independent teammate, who hasn’t had direct contact with the candidate, can help ensure that a fair process is followed.
Before the debrief begins, make sure everyone has submitted written feedback. If you don’t, the people who speak up first or who are loudest are likely to influence, unduly, the opinions of other interviewers. And the whole point of having multiple interviewers is to get multiple independent opinions.
If written feedback is overwhelmingly negative, the moderator should recognize that before the debrief, cancel it, and disposition the candidate accordingly. If you’re running a good process, that shouldn’t happen too often. But the “easy yes” case also shouldn’t happen too often. As Jocelyn Goldfein points out in her excellent writing on interviewing:
High onsite to offer ratios don’t maximize engineering productivity, they maximize interviewer morale. Bringing someone onsite who fails is embarrassing, and mediocre interviews feel like a waste of time no matter what the math says. But avoiding them is an awfully expensive perk.
Collecting written feedback, including numerical ratings focused on specific competencies or attributes, should expose different opinions. Those are a good place to start the discussion. The goal isn’t to reconcile all the differences, but to understand why they’ve occurred.
When all interviewers have had a chance to speak, and to review each other’s feedback, the moderator should solicit yes/no votes. At this point, each interviewer should have a holistic view of the candidate — which they won’t necessarily have after doing one interview focused on a specific competency. Give guidance on where the bar should be set for a “yes”. At Amazon, interviewers are generally instructed to only extend an offer if they believe the candidate will be in the top 50% of current employees. At Intuit, the bar is even higher — the top 25%.
What to do with these votes is company and culture specific. Some companies require a consensus (so any “no” vote is a veto). Some companies require a majority. Many leave it up to the hiring manager. We believe more inclusive decision-making processes are best, where it’s truly the group, not one individual, that produces the offer decision.
You’re not done with the debrief when you make the decision to give an offer! With all the key people who have met the candidate together, you have a unique opportunity to assemble a team plan for convincing the candidate to join. (The fact that your team likes them enough to extend offer means other companies probably do as well — don’t forget about your competition.) Map out who is going to reach out to the candidate and how (phone call? meal? coffee chat?) Don’t be overbearing, but once an offer is extended, don’t go more than a few days without a touchpoint.
Continuous Learning
With every hire your organization makes, you should get better at hiring. As individuals, we do this consciously and unconsciously — when we hire someone who turns out to be a poor fit for the role, we reject the next applicant with same previous employer or major or school. Often, we over-correct or mis-correct.
It’s far better to have a structured process to map new hire outcome data to interview data and analyze the correlations. You can learn about what competencies and feedback signals are actually predictive of job success. And you can learn at scale, in a rigorous and documented way, so a hiring manager’s departure doesn’t mean their hiring knowledge leaves with them.
But all that data mapping and analysis is a heavy burden, even for large organizations. So if you’re looking for help, don’t hesitate to reach out to us at Ansaro :)
