Don’t Add Noise To Hiring Interviews

Everyone I talk to about hiring in a software industry, whether hiring manager or candidate, gets the same look on their face: a mix of anger, disgust, and sadness.

About the laments of job seekers, a lot has been said. It’s a terrible time to be looking for a job, with an unusual amount of competition. Given that’s the case, though, shouldn’t hiring managers have it easy? I talked to one manager who said she had hundreds of applications per day for an engineering role. Shouldn’t that make it easy to pick the perfect one? It doesn’t, and I’d like to dig in to why it’s hard for hiring managers — and what I think we can do to stop making it harder for ourselves.

First, it’s important to remember that hiring has always been awful. It’s a classic situation of a decision being made in a high-noise, low-signal environment.

The cost of a bad hiring decision is enormous: the time-consuming process itself, including countless interviews, then all the training, adaptation, and disruption to relationships (good or bad, it’s still a disruption). Then, add in the opportunity cost of having not made the progress you hoped to with the role — the morale cost of having to start over — and then the literal cost of running another search! A bad hire can cost months of time, and it’s the worst kind of cost — boring, uneducational, and depressing.

And this is why hiring managers are in such an awful position. We have a few hours to make a judgment about a candidate’s ability to perform many complex technical and social tasks. None of these tasks are things that can be demonstrated in a short time frame, so we use proxies as our judgments.

One proxy some people use is the presence of particular universities or companies on a resume. I’ve never understood this; I know too many “Stanford, Ex-Netflix” folks who wouldn’t be great additions to a team (sure, plenty who would, but I don’t want to miss good candidates). This is lazy, and worse, low-signal.

Another proxy used to be an ability to ‘whiteboard’ technical problems. Reversing a linked list, or whatever, was a good proxy for a general computer science education, which used to be — theoretically — a decent metric.

I’m not sure that was ever true, but it’s definitely not any more. We rightfully expect any decent candidate asked to reverse a linked list on a whiteboard to cap the marker, graciously thank their interviewer, and leave. Why? Because we know toy problems are not an ecologically valid proxy for what software engineers do any more. By ‘ecologically valid’ I mean reflecting the work engineers actually do, day-to-day.

And hiring managers know this! They are trying. They want to do a good job, after all; it’s expensive if they don’t. They’re looking for ways to simulate what an engineer does so they can be ‘objective’ and make a good decision. They want to isolate the signals that matter, and toss out the ones that don’t provide information. (After all, the fewer skills you evaluate simultaneously, the better read you get on each. Asking a candidate to juggle knives while discussing an algorithm would be fantastically amusing, and could even result in good hires who would be fun at company parties, but would generate a lot of false negatives along the way. And probably be an OSHA issue.)

So lately I see hiring managers asking a candidate to ‘code something’ in their own work environment, using their own tools. The idea is that a candidate and an interviewer have a conversation about a simple problem, while the candidate types code and explains what they’re doing. The problem is simple, the logic goes, so the candidate can share how they’re thinking. This shows how they communicate, and lets them shine if they’re truly competent. This should focus on ‘the skill’ that matters, right? Writing software?

This sounds reasonable! Get an engineer talking about something they should be able to understand. Use code as the subject of the conversation, since coding is a big part of what we do. And this can work. But it’s only halfway reasoned, and it throws out valuable signal.

The problem is that this is far from ecologically valid. In cognitive linguistics research, there is a thing called a verbal interference task. The idea is that by asking someone to narrate or otherwise recite words while performing another task, you can cripple some of their cognitive functions selectively. Communication is critical, but engineers don’t communicate while they are thinking. This is deeply unnatural and disruptive, which is cruel, but more importantly, it adds noise.

An interview is already a high stakes conversation, and inherently stressful. Adding more complex tasks — narrating something that’s already very complex while trying to generate words in another modality simultaneously (typing) — is a great way to destroy the signal you want to judge. If this sounds to you like a terrible thing to include in an interview, you’re right.

So it’s kind of shooting ourselves in the foot. But back up. Can we focus on the intuition about being more ecological? What are the skills that matter here, and can we highlight those, removing both noise and low-signal tasks?

Well, one thing I don’t care about is ‘typing’. Why not assume anyone interviewing for the role can operate a keyboard or otherwise get code into a text file on demand. How could we do that?

Take a careful look at the interviewer. They’re mostly listening. They’re comfortable, in a safe role, taking care to make the candidate comfortable and ask smart questions, but they are unstressed and somewhat scripted. They have cognitive ‘slack’ to spare.

The conclusion sounds radical, but really it’s familiar to anyone who has done some pair programming: code, but have the interviewer do the typing.

This lets us as interviewers do the ‘junk’ work that is uninformative, while directing the most attention to the candidate’s ability to communicate and reason — the most signal-dense elements of the interview.

Practically, how does this work?

The candidate directs the action. If they are leaving out details, we can ask for them. Once a skill is demonstrated, we can make some assumptions and ‘fast-forward’ to more complex parts of the problem, directing the conversation to where it is illuminating.

As interviewers we can select problems that are scalable — most of the modern interview questions can work. A well-selected problem can be quantifiably gradable, even, if that’s desired, while still affording skilled candidates time to digress, discuss optimizations or details, or reveal elements of their coding philosophy that can show a great deal about what they would be like to collaborate with. By picking a problem and coding it up ahead of time, we can be even more comfortable coding it again, freeing yet more attention for value-dense conversation.

The end result is that an hour-long coding session can produce a tremendous amount of insight about a candidate — their coding skills, flexibility, emotional maturity, and communication.

I’ve used a technique based on this analysis for years. It’s hard to judge the efficacy of a hiring process because information about false negatives is hard to come by, but I’ve kept records. They show this as having a very high predictive index on future performance — and revealing a great deal about a candidate’s collaborative and communication abilities very quickly, by making them more comfortable while allowing the interviewer more control to dynamically investigate elements they feel they need more detail on.

Never yet have I hired a candidate who could talk cogently, abstract, model systems well, deal with ambiguity, and answer detailed questions about alternatives, but who was secretly unable to type.

So, cut out the typing. It takes a bit of prep work, and training. But it lets us as interviewers get far more signal-per-hour.