Are resumes better than a coin flip? [Lerner]

Itamar Goldminz
Nov 15, 2016 · 4 min read

Aline Lerner, a master recruiter, decided to run an interesting experiment:

She created an anonymized data set of 51 resumes, 33 belonging to “strong” candidates, and 18 to “less strong” candidates. Classification was determined based on how the candidates fared in actual job interviews and subsequent job success.

A group of 152 experts (22 in-house recruiters, 24 agency recruiters, 20 hiring managers, 86 engineers involved in the hiring process) was each shown a random sample of 6 resumes from the data set and asked a simple question: “would you interview this candidate?”

Answers were codified with “yes” = 1 and “no” = 0. These were the results:

Image for post
Image for post
Image for post
Image for post

On average, participants guesses correctly, 53% of the time, barely better than flipping a coin (in-group averages ranged from 48% to 56% but the differences were not statistically significant).

While the experiment may not be as scientifically rigorous and we’d like it to be, it’s the most diligent attempt that I’ve seen to test out a hypothesis that I’ve been contemplating myself for quite a while:

A typical hiring process consists of two sub-processes running in parallel. The emphasis on each changes from one step to the other and from one company to the other:

  1. — meant to give the company confidence that this is the candidate they want to hire
  2. — meant to give the candidate confidence that this is the company they want to work for

Resumes are the initial filter for #1. So much time and energy is invested in writing them (by the candidate) and screening them (by the recruiter/hiring manager), but the above experiment suggests that a coin flip would be a more effective initial screener. And we haven’t even started talking about the biases that they introduce to the screening process (plenty of examples here, just Google it).

Is there a better alternative?

The short answer is yes. It’s called a “work sample test” and the sad part is that it was proven to be superior back in 1998, almost 20 years ago, while very little has changed in the way we screen candidates since.

A work-sample test simply means giving candidates a sample piece of work to the one they’ll be asked to do in their role. For software engineers, it doesn’t have to be a coding project, it can also be multiple-choice quiz like the folks at Triplebyte use.

I’m sure many of you have some strong push-back arguments. I know I did when I first started thinking about this. Let me address some of them in FAQ-form:

Creating a work-sample test sounds like a lot of work. Is it really worth it?

Consider the alternative cost: both the time investment in (meaningless) resume screening, and losing out on good candidates due to an ineffective screening process.

I see how a work-sample test works well for engineers, but can it work for other roles as well?

Absolutely. One of my favorite examples is one of my old bosses screening Executive Assistant candidates by asking them to create a fake itinerary for him given a set of constraints. I once had to conduct a test phone call for school operations role with a fake parent whose kid was consistently late for school. Hiring sales people? Have them videotape themselves giving a 10 min pitch of the product they are currently selling.

What about cheating?

Some forms of test are in theory prone to cheating. But so do resumes: we typically only check references in the last step of the process, after we’ve already made the investment of having the candidate go through the entire interview process. If you’re really paranoid about cheating: you can have the candidates come on-site to take the test, or do it live — but keep in mind that you’re considering a replacement to a “coin flip” process, a minor cheating risk seems like a reasonable trade-off.

Completing a work-sample test seems to be rather time consuming. Is it fair to ask candidates to invest so much time up-front in their candidacy?

Fair. A work-sample test don’t have to be a lengthy exercise. It can also be a 10–15 min assignment, if you know what you’re doing. For lengthier test consider: a) time-boxing b) paying candidates for their time. Sounds expensive? more expensive than using your current “coin flip” screening process?

We’re a small startup, and there’s no way candidates will be willing to invest the time in taking a work-sample test to work for us. What can we do?

If you’re a small startup, the thing you really can’t afford is having a screening process that rejects about 50% of the qualified candidates that are interested in working for you. If you’re convinced that you can’t use a screening filter up-front — get rid of it altogether and just use a jobs@ email address or a contact info form instead. Give anyone who expressed interest a call.

Org Hacking

Solving Human Puzzles

Sign up for OrgHacking Newsletter

By Org Hacking

Solving Human Puzzles Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Itamar Goldminz

Written by

I enjoy solving human puzzles

Org Hacking

Solving Human Puzzles

Itamar Goldminz

Written by

I enjoy solving human puzzles

Org Hacking

Solving Human Puzzles

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store