The Interview Black Hole
Visit us at https://www.ansaro.ai to learn more about how we’re using data science to improve hiring
I remember my worst interview like it was yesterday. I was on the phone with the interviewer, who we’ll call “Mike”. He started with a convoluted math problem. After 3 failed attempts, I was already convinced I wasn’t getting the job and I asked if we could just move on. Nope. After another 3 failed attempts, we sat in silence until our scheduled 30 minutes elapsed and Mike hung up.
The next day, HR called me, said Mike had come away with a “terrific impression”, and invited me to interview again. Bad interview experiences like this, that leave the candidate feeling baffled and the interviewer with no quantifiable information, are all too common. But interviews are incontrovertibly an important — often the most important — part of the hiring process. They take many formats: phone screens, in-person interviews, whiteboarding sessions, case studies, “fit” interviews, panels, group challenges, and more. Whatever the format, all interviews ultimately serve two purposes:
- For the prospective employer, the interview is an opportunity to evaluate a candidate’s knowledge, skills, abilities, and experiences in terms of the future demands of the role. Will they be able to perform the job well?
- For the candidate, the interview is a chance to prove they’re the best person for the job, and evaluate the company and its employees.
Job interviews aren’t going away any time soon, so let’s dive into how we can make sure they’re productive and (hopefully) tolerable for both the interviewer and the candidate.
Structural changes
Decades of research [1, 2, 3] has shown that structured interviews are among the best predictors of job performance and best ways to reduce bias in the hiring process. [4] But what exactly is a structured interview?
By structured interviews, we mean 3 things:
- Use a consistent set of questions across candidates applying for the same job, and ask them in the same order.
- Use questions that relate to the abilities needed for the job. Past behavior is the best predictor of future behavior, so ask about prior relevant experience. You also want to understand whether a candidate will respond appropriately in new circumstances, so ask about hypothetical on-the-job situations.
- Evaluate responses on a standard scale, for example, a 1–5 rating for each competency assessed. Even better is to anchor those ratings to descriptive evaluations to maintain consistency across both candidates and interviewers.
When planned well, structured interviews also make for a much better candidate experience:
- Candidates don’t get asked the same questions over and over again,
- There’s a finite number of interviews, so candidates don’t feel like they keep getting hauled back in with no end in sight,
- And candidates are more likely to feel that they were fairly evaluated.
How does interview data drive hiring decisions at your company?
At many companies, including those who have partnered with Ansaro, no interview feedback is collected in a digital form or systematically saved, so the information disappears into the “interview black hole”. While interviews are the critical drivers of decision-making in almost all hiring processes, companies are consistently failing to capture any data that reflects why and how those interview-driven decisions are being made.
At some companies, candidate feedback is summed up with a single, subjective “thumbs up / thumbs down” rating. These ratings are at least saved centrally in the applicant tracking system (ATS), but with the minimal amount of information you get from such a simple rating, you might as well be evaluating the candidate with a coin flip. Collecting better data from your structured interviews will not only let you compare all the candidates for one job, but it will help you learn about your interview process (“Are we asking the right questions?”, “Are the right people serving as interviewers?”) and make the best decision about a future hiring need. At a minimum, you should record:
- Who performed the interview,
- What competencies, skills, and topics were covered,
- Assessments of the candidate for each topic using a standardized, quantified scale,
- Any freeform feedback on each topic,
- [Optional] and the interviewer’s overall assessment of the candidate (or alternatively, compute an overall score from the topic scores).
Evaluating the evaluators
Training interviewers to conduct better interviews has been shown to make them more effective at evaluating candidates. [5] But what if you don’t have the time or resources for extensive training? Once you’ve begun driving hiring decisions with data from structured interviews, you can close the loop by using that data to figure out who your most effective interviewers are. Who always says “no”? Who tends to pick the best performers? With that information, you can establish more effective interviewing strategies for your company.
We’ll leave you with advice about interviewing from Nobel prize-winning psychologist Daniel Kahneman, from a chapter on using formulas over intuition: [6]
First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on). Don’t overdo it — six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1–5 scale. You should have an idea of what you will call “very weak” or “very strong.”
These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate, add up the six scores. […] Firmly resolve that you will hire the candidate whose final score is the highest, even if there is another one whom you like better. […] A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as “I looked into his eyes and liked what I saw.”
–Matt, Cofounder, https://www.ansaro.ai
matt.mollison@ansaro.ai
We’re hiring! If you’d like to help companies make the best hiring decisions, check out our job postings: https://angel.co/ansaro/jobs
References
- Campion M.A., Palmer D.K., & Campion J.E. (1997). A review of structure in the selection interview. Personnel Psychology, 50, 655–702.
- Chapman, D.S. & Zweig, D.I. (2005). Developing a nomological network for interview structure: Antecedents and consequences of the structured selection interview. Personnel Psychology, 58, 673–702.
- McDaniel M.A., Whetzel D.L., Schmidt F.L., & Maurer S.D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599–616.
- Interestingly, a recent re-evaluation of employment selection research shows that unstructured interviews may not be as bad as previously believed. Oh, I.-S., Postlethwaite, B.E., & Schmidt, F. L. (2013). Rethinking the validity of interviews for employment decision making: Implications of recent developments in meta-analysis (Chapter 12, pp. 297–329). In D.J. Svyantek & K. Mahoney (Eds.), Received wisdom, kernels of truth, and boundary conditions in organizational studies. Charlotte, NC: Information Age Publishing.
- Dougherty T.W., Ebert R.J., Callender J.C. (1986). Policy capturing in the employment interview. Journal of Applied Psychology, 71, 9–15.
- Kahneman, D. (2011). Thinking, Fast and Slow. New York: Farrar, Straus, and Giroux.
