It’s Time to Kill the Informal Interview

It starts innocently enough:

“I hear you’re looking for a new position. We’re hiring. Want to grab coffee and chat?”

Unfortunately, it leads to systematically bad hiring decisions.

I hope you’ll hear me out for a second, because the evidence here is really clear, and yet it’s still ignored by the vast majority of companies.

Can we please stop? There’s a better way.

The Problem: Over-Weighting

We’ve known for years that interviews don’t work. That is, candidates that interview well don’t, on average, perform better on the job. Interviews are a shitty signal.

But, it’s worse than that. Decades of research into job performance has demonstrated that candidates that interview well, perform worse.

I’ll say that again. Good interview = bad hire.

How can that be?

The issue is that employers consider a number of factors when making a hiring decision. Years of experience, GPA, references… and what kind of shoes the candidate wears to the interview.

Good-interviewers perform worse than the bad-interviewers because, when someone interviews well, we tend to overlook weakness in other areas. (“I know she’s inexperienced, but she has such great energy!”)

On average, the bad-interviewers must have looked stronger on other factors to get hired. Turns out those other factors are better indicators of job performance.

Said another way, we overweight the interview in the hiring decision.

The problem has two parts. First, informal interviews don’t provide much good information — they’re primarily a test of conversation skills, and jobs usually involve far more than that. Second, we consistently overestimate our own abilities to predict job performance from a casual conversation.

The second reason is the scary one, because it’s difficult to counter. No matter what the research tells us, we continue to believe that we can judge people based on an interview.

And interviews do provide some real information. You might actually learn something from a candidate’s footwear, how much they smile, and how much they say “I” instead of “we.” We can probably screen out some of the assholes, even with an unstructured interview.

But, once we’ve interviewed someone, it’s almost impossible to discount the opinion we’ve developed from that interview, despite the fact that our opinion is based on almost no real information. Availability bias and confirmation bias are nearly impossible to avoid.

From my own experience, the interview process typically counts for at least half of most companies’ hiring decisions, after an initial screen. Often it’s 90-100%. Given how poor the data from most interviews is, they should probably only account for 10 or 20% of the decision.

There’s No Substitute For Time in the Trenches

We may not be able to fix our biases, but we can certainly improve the quality of data we’re using.

Having made a bunch of bad hiring decisions, I’ve found the only truly meaningful information about a candidate comes from working with them. You just can’t tell from an interview what it will be like to work with someone — the intangibles that matter don’t correlate well with interview skill. There’s no substitute for time in the trenches together.

So the best way to make good hiring decisions is to hire people you’ve already worked with. You can get further doing this than most people think. Work harder to recruit the people you know are good, and waste less time getting coffee with people you don’t know.

Once you’ve exhausted your own list of former co-workers, you can hire the people that they’ve worked with. They have good data, too. You’re one person removed from that information, so the signal is a little less clear, but it’s infinitely better than interviewing randos.

Now let’s say assume you need to hire beyond your network, either because you’re bringing in skills outside your network, or in a new location, or because you just need to hire a lot of people. The next-best source of information is references.

References are people who have worked with the candidate, so they have good information on how they work. The challenge in hiring is to extract that information. References are highly likely to BS you, so you need a script of probing questions to try to get past the fluff. Make them force rank the candidate against their peers. Find “back-door” references — people the candidate has worked with that aren’t on their reference list. That’s what LinkedIn is for. And spend more than 5 minutes on the phone—the best intel usually comes right before you hang up. Let them talk a while.

I’m astounded by the fact that most employers use references as no more than a negative screen. After the company has already made a decision to hire, someone in HR makes a 5 minute call to their references. Pointless.

I believe that in many cases we’d actually hire better if we didn’t interview at all, and just called references instead. The data is so much better, and we wouldn’t be tempted to overweight our own opinion from the interview, which is as likely to be wrong as right.

I’m not likely to win that battle — candidates do typically want to meet the people they’ll be working with, afterall.

So let’s see if we can improve the interview so it generates real information.

More Structure Means Better Interviews

Technical interviews, done right, yield good information. That’s because a technical interview simulates the experience of working with someone.

For an engineer, you can give them an engineering problem to solve. For a strategy role, you can give them a case study. For an ops, give them a nasty management problem and ask for a solution.

The key is to give every candidate the same problem. Script your questions and stick to the script. That way you can compare performance.

I prefer “homework assignments” which give the candidate time (a couple hours, typically) to do some work and show work product. The main reason is that this is closer to what work actually looks like — people do some work alone, and generate code or an email or slide deck that explains the output of their work. Homework assignments are a better simulation of real work than brainstorming on a whiteboard.

If homework isn’t possible, and case study interviews don’t work, you can at least script your interviews. Again, ask all candidates the same questions so you can compare their answers. Know exactly what skills you’re looking for and ask questions that probe for those specific skills.

Structured interviews are more work to develop and run, of course, but don’t forget that, without structure, interviews are worse than worthless. Startups like Triplebyte are building great businesses handling initial technical interviews for you. I expect we’ll see similar services extend beyond software development, too.

Job-Seekers: Don’t Let Them Trick You

This matters just as much if you’re job-hunting.

If you get invited to coffee to “chat” about a job, by all means, accept. But don’t let them get away with an unstructured interview. If you do, they’ll use it to make a bad hiring decision.

Instead, turn that informal interview into a structured one. Start with probing questions about their needs. Find out what exactly they’re looking for in people they want to work with.

Then verify with them that you’ve got it right: “So, it sounds like you’re looking for A, B, and C. Does that sound right?”

Now you’ve given the interview the structure it needed. More than that, you’ve given them a framework by which to evaluate you.

Next, sell yourself. Tell them about specific experience that you have that matches their needs.

If that feels like a strange thing to do in a casual conversation, you can tell them exactly what you’re doing. Say, “hey, do you mind if I ask some more questions about what you’re looking for? I’d love to tell you a little about my experience but want to make sure I talk about the things that matter to you.” I can’t imagine anyone objecting to that.

After the interview, you can send them some homework, even if they didn’t ask for it. Shoot over some links to relevant projects you’ve worked on, or a thoughtful follow-up email that shows what your work product looks like.

Show them what it’s like to work with you. Get them the good data.

Can We Still Have Coffee?

There’s still a time and place for a casual chat. But, if you’re looking to hire, it’s important to recognize that casual conversations are for recruiting, not for evaluating candidates. You’re selling them on the company, and you’re qualifying the lead, making sure there’s enough interest on both sides to invest in a real interview process. You’re not yet evaluating them for the job.

Informal interviews, by themselves, are worse than a waste of time, they’re creating massive inefficiencies in the labor market.

It’s time to put them to rest.