Design flaw of interviews

I have mixed feelings about interviews. As a candidate, I’m always frustrated and stressed out about interviews. I perform quite poorly on formal interviews. As an interviewer, I’m under pressure to judge person’s future performance based on 45–50 minutes of face-to-face time. That’s a hard thing to do. There’s a classic school of interview process using a whiteboard to solve an algorithm problem, and over the years I’ve heard a lot of feedbacks from candidates that interview performance doesn’t reflect their skills, talents and work performance.


Single whiteboard interview session is usually about 45–50 minutes long, and it’s split between two activities: two algorithmic challenges, an algorithmic challenge, and testing. A system design usually takes the whole session. That means that a candidate usually limited to 25–30 minutes to understand the problem, find a solution, optimize the solution and implement the solution. A candidate uses a whiteboard or a shared online notebook to go through problem-solving and to write code. Interviewers judge candidate’s performance and they expect a candidate to demonstrate:
 — proficiency in algorithms
 — proficiency in data structures
 — problem-solving skills 
 — code style/implementation

Sounds straightforward and fair, right. The problem is: it’s only on paper. A candidate put in an environment that doesn’t match working conditions. Companies hire people to do a job well, but their interviews only check how well a person has prepared for the particular interview process.

In my entire 13-years working career, I have never been in a position to do my job under conditions of a whiteboard interview. I either had a tricky problem that required a smart choice of algorithms and data structures and had plenty of quiet time to think it over; or it was an incident, when blazing-fast troubleshooting, communication, and quick-and-right decisions were critical. The whiteboard interview doesn’t mimic either of them. You know when last time I was under pressure to solve a problem under certain time besides interviews? In a college.


All that made me think of what would be a good way to find an engineer. If I would hire a software engineer, what qualities I would expect.

— Ability to understand different domain problems and get to the details/ask right questions.
 — Ability to come up with a technical solution to a domain problem.
 — Ability to clearly communicate/explain their solution.
 — Ability to accurately implement the solution.
 — Follow good practices/be disciplined.
 — Ability to verify the solution.
 — Ability to read/understand somebody else’s code.
 — Ability to identify a problem in a code/product/process and explain it concisely.
 — Ability to prioritize work based on complexity/value/urgency of a future deliverable.
 
People rarely work in a vacuum of a 1-person team; they have to be able to collaborate with other team members and other teams. Then from any team member, I would expect following qualities:
 — a shared understanding of goals
 — fit into the team’s dynamics
 — ability to listen to feedback and apply it
 — ability to vouch for somebody else’s idea
 — communicate respectfully and friendly to all colleagues, regardless of age, gender/sexual orientation and race
 — trust the team to do their best (aka don’t think that everybody before them were idiots)
 — not being political:
 — prioritize work based on work, not where it came from
 — voice their opinion based on their understanding, not on a person they’re talking to

Let’s see what qualities of a future employee whiteboard interviewers check:

— Ability to understand different domain problems and get to the details/ask right questions. If it’s not a system design session, then problem usually small enough (to be solvable in 25–30 minutes) that there’s not much of a domain attached to check the quality. Check for system design session, uncheck for algorithms.
 — Ability to come up with a technical solution to a domain problem. How to translate a real-world problem to a technical solution? If it’s not a system design session, then a problem is formulated in terms that don’t check the quality. Check for system design session, uncheck for algorithms.
 — Ability to clearly communicate/explain their solution. Proponents of whiteboard interview claim that slower writing forces candidates to communicate their thought process. I don’t agree with that but won’t argue much. Check.
 — Ability to accurately implement the solution. Check.
 — Follow good practices/be disciplined. I don’t think it’s possible to demonstrate the quality. If a candidate is very sloppy, then interviewer could notice that, but beyond that, it’s impossible to judge the quality. Uncheck.
 — Ability to verify the solution. Check.
 — Ability to read/understand somebody else’s code. There is no time to prove that. Uncheck.
 — Ability to identify a problem in a code/product/process and explain it concisely. There is no time to prove that. Uncheck.
 — Ability to prioritize work based on complexity/value/urgency of a future deliverable. There is no time to prove that. Uncheck.

There’s no time to see teamwork, and team aspect is ignored completely. On paper, whiteboard interviews look not bad: a combination of algorithm-heavy problems, system design, and testing sessions check off 67% of the qualities I marked for a good engineer.

If you think about that a bit longer, then in a context of real work, it’s not very relevant. The whiteboard interview is quite hostile and stressful for candidates, and it’s not a very accurate way of predicting future employee performance. It checks for a limited and unbalanced number of qualities, it puts an unusual stress and asks a lot from a candidate. It says nothing about a candidate as a team player. The interview process includes more and more tricky algorithmic problems to compensate for a poor accuracy. Recruiters explicitly say to candidates, that average successful candidate spends 40–60+ hours to prepare for the interview. That’s a tremendous commitment for people who are already working a full-time job. It appears to me that the process is optimized to be easier on interviewers and be conservative with positive results.

Somebody may say that a whiteboard interview is a “lab test” when a candidate is in a stressful situation, and interviewers want to see how a person behaves under stress. I don’t think this exercise correlates with actual work performance as a software engineer. There’s place in a world for jobs that need stress resilience, mental and physical. I don’t think software engineering is one of them. Moreover, if a company purposefully creates a stressful environment during an interview, I would consider that a red flag.

My claim is that interview process in its current form doesn’t find “best engineers,” it finds “best candidates.” I’ve witnessed myself situation when a candidate easily passed a whiteboard interview, and a hiring manager was thrilled until they discovered that the person was not quite a team player and was toxic for the team and team’s morale and performance degraded. The whole team breathed out when that person left the company to pursue new endeavors.


If a whiteboard interview is not good, then what would be the hiring process? I believe that a combination of pre-interview call with a hiring manager, a homework assignment, and an on-site workshop is best way to hire. Let’s see

1. Make sure that overall qualifications and goals of a candidate match to the team’s needs and goals.

Before the technical screening/interview, have a phone call between a candidate and a hiring manager. 20–25 minutes will be enough to understand if both sides want the same. A call with recruiter usually doesn’t work that well, unless recruiter is embedded into the team and knows team goals/needs from inside.

2. Have a homework assignment.

Give a project template with a bug and ask to fix it. Share a small project and give three tasks and ask to prioritize. Set a reasonable deadline.

3. On-site workshop.

Invite candidate to work on-site for a day with an engineer(s) from the hiring team. The important point is that it must be collaboration, not an interview. It could be a project from the same set as homework, but important part is that team engineer wouldn’t know the solution.

Let’s see how it stacks up against “a good engineer” criteria.

— Ability to understand different domain problems and get to the details/ask right questions. Homework + workshop.
 — Ability to come up with a technical solution to a domain problem. Homework + workshop.
 — Ability to clearly communicate/explain their solution. Workshop.
 — Ability to accurately implement the solution. Homework + workshop.
 — Follow good practices/be disciplined. Homework + workshop.
 — Ability to verify the solution. Homework + workshop.
 — Ability to read/understand somebody else’s code. Homework or workshop.
 — Ability to identify a problem in a code/product/process and explain it concisely. Homework or workshop.
 — Ability to prioritize work based on complexity/value/urgency of a future deliverable. Workshop.
 
qualities of a good team member
 — has a shared understanding of goals. Hiring manager phone call.
 — fit for the team. Workshop.
 — ability to listen to feedback and apply it. Workshop.
 — ability to vouch for somebody else’s idea. Workshop.
 — communicate respectfully and friendly to all colleagues, regardless of age, gender/sexual orientation and race Workshop.
 — trust the team to do their best (aka don’t think that everybody before them were idiots)
 — not being political:
 — prioritize work based on work, not where it came from
 — voice their opinion based on your understanding, not on a person you’re talking to

It covers 100% of qualities of a good engineer, some of the qualities are covered twice by homework and a workshop. Plus 80% of qualities of a good team member. The design of homework and workshop assignments is critical in the assessment of candidates, and some companies came up with criteria for these types of exercises, and I believe it should be a standard.