Are resumes better than a coin flip? [Lerner]

Aline Lerner, a master recruiter, decided to run an interesting experiment:

She created an anonymized data set of 51 resumes, 33 belonging to “strong” candidates, and 18 to “less strong” candidates. Classification was determined based on how the candidates fared in actual job interviews and subsequent job success.

A group of 152 experts (22 in-house recruiters, 24 agency recruiters, 20 hiring managers, 86 engineers involved in the hiring process) was each shown a random sample of 6 resumes from the data set and asked a simple question: “would you interview this candidate?”

Answers were codified with “yes” = 1 and “no” = 0. These were the results:

On average, participants guesses correctly, 53% of the time, barely better than flipping a coin (in-group averages ranged from 48% to 56% but the differences were not statistically significant).

While the experiment may not be as scientifically rigorous and we’d like it to be, it’s the most diligent attempt that I’ve seen to test out a hypothesis that I’ve been contemplating myself for quite a while:

Do resumes really matter?

A typical hiring process consists of two sub-processes running in parallel. The emphasis on each changes from one step to the other and from one company to the other:

  1. A screening process — meant to give the company confidence that this is the candidate they want to hire
  2. A selling process — meant to give the candidate confidence that this is the company they want to work for

Resumes are the initial filter for #1. So much time and energy is invested in writing them (by the candidate) and screening them (by the recruiter/hiring manager), but the above experiment suggests that a coin flip would be a more effective initial screener. And we haven’t even started talking about the biases that they introduce to the screening process (plenty of examples here, just Google it).

Is there a better alternative?

The short answer is yes. It’s called a “work sample test” and the sad part is that it was proven to be superior back in 1998, almost 20 years ago, while very little has changed in the way we screen candidates since.

A work-sample test simply means giving candidates a sample piece of work to the one they’ll be asked to do in their role. For software engineers, it doesn’t have to be a coding project, it can also be multiple-choice quiz like the folks at Triplebyte use.

I’m sure many of you have some strong push-back arguments. I know I did when I first started thinking about this. Let me address some of them in FAQ-form:

Q: Creating a work-sample test sounds like a lot of work. Is it really worth it?

A: Consider the alternative cost: both the time investment in (meaningless) resume screening, and losing out on good candidates due to an ineffective screening process.

Q: I see how a work-sample test works well for engineers, but can it work for other roles as well?

A: Absolutely. One of my favorite examples is one of my old bosses screening Executive Assistant candidates by asking them to create a fake itinerary for him given a set of constraints. I once had to conduct a test phone call for school operations role with a fake parent whose kid was consistently late for school. Hiring sales people? Have them videotape themselves giving a 10 min pitch of the product they are currently selling.

Q: What about cheating?

A: Some forms of test are in theory prone to cheating. But so do resumes: we typically only check references in the last step of the process, after we’ve already made the investment of having the candidate go through the entire interview process. If you’re really paranoid about cheating: you can have the candidates come on-site to take the test, or do it live — but keep in mind that you’re considering a replacement to a “coin flip” process, a minor cheating risk seems like a reasonable trade-off.

Q: Completing a work-sample test seems to be rather time consuming. Is it fair to ask candidates to invest so much time up-front in their candidacy?

A: Fair. A work-sample test don’t have to be a lengthy exercise. It can also be a 10–15 min assignment, if you know what you’re doing. For lengthier test consider: a) time-boxing b) paying candidates for their time. Sounds expensive? more expensive than using your current “coin flip” screening process?

Q: We’re a small startup, and there’s no way candidates will be willing to invest the time in taking a work-sample test to work for us. What can we do?

A: If you’re a small startup, the thing you really can’t afford is having a screening process that rejects about 50% of the qualified candidates that are interested in working for you. If you’re convinced that you can’t use a screening filter up-front — get rid of it altogether and just use a jobs@ email address or a contact info form instead. Give anyone who expressed interest a call.