How to create code assessments and interviews that developers won’t hate

Un-cracking the code interview

It’s become a cliché in the industry: the technical interview is broken. Virtually everyone agrees that most technical hiring processes are biased, burdensome, and embarrassingly ineffective at identifying the best candidates for the role.

And yet, despite widespread condemnation, it seems nothing ever changes. Why? Simple: supply and demand. Pundits and industry leaders that ignore this reality may have good intentions, but they aren’t actually helping us find a better path forward.

Why code assessments and interviews need to exist

The reality is that there are more senior roles than senior candidates, but fewer junior roles than junior candidates. This mismatch isn’t that surprising – thousands of developers graduate computer science programs and coding bootcamp every semester. Employers typically receive thousands of candidates for each job posting.

In some imaginary world there’d be enough roles for every developer, and developers would only apply to the roles they qualify for. The interview would be a ceremonial process to confirm the meritocratic match, and no one would complain again. But that obviously isn’t reality. In fact, nearly the opposite is true: the majority of people apply to jobs they’re not quite qualified for and employers have limited time to meet every candidate.

That’s why employers deploy a combination of automated and manual capabilities in an attempt to filter out unqualified candidates and identify potential fits.

The intention is (usually) pure! Employers want to give every candidate a fighting chance to land the job. The alternative would be far worse and more biased, singularly favoring candidates with recognized brands and universities on their résumés.

We must accept that automation needs to exist for recruiting to scale. Any company, industry leader, or pundit that says otherwise is being disingenuous and likely giving preference to highly privileged candidates from their network. Expose them by asking how employers are supposed to give every candidate a remotely fair shot.

That being said, are automated code assessment capabilities perfect? Absolutely not. Unfortunately they can even be counterproductive, repelling good or disadvantaged candidates in favor of desperate ones. Worse, these processes are sometimes reinforced by so-called experts and software platforms designed for high-velocity recruiting rather than high-quality recruiting.

Let’s dig into why and then determine a better path forward.

Why code assessments and interviews usually suck

Google notoriously pioneered riddle-based interviews. Thousands of companies built recruiting processes and selected code assessment capabilities mimicking Google only to find out later that such questions are useless. In an interview with The New York Times, Google’s SVP of People Operations, Laszlo Bock came clean: “On the hiring side, we found that brainteasers are a complete waste of time. They don’t predict anything. They serve primarily to make the interviewer feel smart.”

Or look at HireVue, a software platform which claimed it could use artificial intelligence and biometric data to evaluate candidates. As one AI researcher put it to The Washington Post: “It’s a profoundly disturbing development that we have proprietary technology that claims to differentiate between a productive worker and a worker who isn’t fit, based on their facial movements, their tone of voice, their mannerisms. It’s pseudoscience. It’s a license to discriminate.”

HireVue halted use of the application after complaints were filed with the FTC, but by then more than one million candidates had already been interviewed through the horrific process.

Again though, despite all the problems with today’s technical interviews and assessments, the alternative — a world in which they didn’t exist — would probably be even worse.

If companies didn’t use automation in the screening process, there’d be even more bias. An unfair advantage would be given to candidates that embellish their résumés, candidates from privileged backgrounds and prestigious universities, or candidates with great personalities but suspect qualifications.

Assessments should exist! They just shouldn’t (and don’t have to) suck.

While no candidate will ever adore an interview or assessment, there are mistakes that any employer can avoid to ensure the hiring process is as scalable, unbiased, and painless as possible.

How to improve code assessments and interviews

I’ve spent years trying to fix the technical interview process by leveling the playing field. With my platform, Coderbyte, 750,000 developers have prepared for interviews and 1,500 employers have interviewed and hired great candidates.

We’re famously principled about our product roadmap, rejecting requests for features that would ruin the candidate experience or otherwise bias the interview process. We frequently ban bad actors (both employers and candidates) from our platform because sustainable technical interviews are a collective effort. Suffice to say, we’ve learned a lot about how to make technical interviews and assessments better. Our recommendations may be counterintuitive at first, but they’re rooted in data from millions of automated assessments and interviews conducted on our platform.

Recommendation #1: Keep it short

Mistake: It’s compelling to make your code screening assessments as robust as possible with coding challenges and questions to evaluate every skill listed on the job description. If you’re one of the top technology companies in the world, maybe you can get away with it. But it doesn’t work for virtually every other company and here’s why:

• No developer likes coding in an assessment platform outside of their preferred environment, and they’ll grow increasingly frustrated and likely to make silly mistakes with longer assessments.
• Top candidates won’t have the time to take longer assessments because they’re being recruited by many other companies, some of which will be willing to bypass the screening phase altogether. The opposite is true for the worst candidates who will have plenty of time to spend taking assessments.

Solution: Even a very straightforward coding challenge will filter out 60%-80% of unqualified candidates, which is an ideal target range. Any higher than that and you’re probably inadvertently filtering out more good candidates than bad candidates. The benefit of screening for additional skills is thoroughly outweighed by the diminishing returns of lengthy assessments. Drop-off rates begin increasing once your assessment reaches 30 minutes, and dramatically so after 60 minutes at which point the assessment is completely counterproductive. Screening assessments aren’t meant to be perfect — they accelerate prioritization but your team still needs to conduct actual in-person or remote interviews.

Recommendation #2: Keep it unbiased

Mistake: Your organization solves unique technical challenges with a specific stack, so it’s enticing to create custom assessments, challenges, and questions to test candidates for the requisite skills. Further, your engineering team searched the internet and found solutions to the challenges from your screening platform’s library.

The fact is that motivated candidates are always going to find ways to cheat, and there are better solutions to prevent cheating (covered in the next section) than creating custom challenges. Further, most custom challenges create more problems than they solve:

• Creating unbiased assessments is truly difficult and the only way to really test them is to do so at scale and get feedback over a long period of time from both candidates and hiring managers. 10 years of engineering experience does not qualify your VP of Engineering to create assessments because it’s a completely different skillset. Our challenges on Coderbyte have been iterated on for almost a decade based on feedback from millions of developers. Issues related to readability, test cases, edge cases, and time constraints have been ironed out. The carefully crafted challenge that your engineering team thinks is so novel probably doesn’t work well in an automated screening environment where the candidate can’t ask questions to get clarity.
• Consider just a few common mistakes you’ve probably already made. You may have asked questions that assumed common knowledge like how many innings are in a baseball game, but this knowledge may not be so common amongst people from various backgrounds who are now at a disadvantage in your interview process. Or your team may only be looking for a specific answer having never considered an alternative but equally acceptable answer which is then grading wrong either automatically or in an interview.

Solution: If your team insists on using custom coding challenges, at least ask someone else from the engineering department to independently solve it just once. We have found that 9 out of 10 custom challenges used on Coderbyte can’t be passed by an engineer within the hiring team! Just a few iterations can dramatically reduce bias in a custom challenge or question. That’s why we have a mandatory expert review of all custom challenges instead of the industry standard laissez faire approach that leads to terrible candidate experiences.

Recommendation #3: Keep it realistic

Mistake: Cheating and bad actors are unfortunately rampant in all forms of standardized testing and credentialing, so it’s reasonable to want to mitigate cheating on coding assessments and interviews! The issue is that many of today’s most common ‘cheating detection’ mechanisms are lazy and poorly considered. They lead to assessments experiences that are frustrating and unrealistic, not to mention discriminatory and unethical:

• Tools like HackerEarth offer ‘advanced proctoring’ which creates an invasive and completely unrealistic coding environment for candidates. Candidates have to turn their webcams on to ensure it’s really them, while basic functionality like copy and paste or navigating the browser are blocked. The whole point of assessments is to make hiring as unbiased and meritocratic as possible, but webcam recordings directly invite discrimination. Further, self-respecting developers will feel uncomfortable participating in such an assessment, while desperate ones can still easily cheat by simply having a second computer or phone right next to them.
• Most cheating detection techniques will reflect poorly on your employer brand, while repelling qualified candidates and inviting sophisticated bad actors. If cheating is so rampant amongst your candidates, you might want to reconsider your channels for sourcing. Regardless, there are superior approaches to detecting cheating and identifying strong candidates.

Solution: At Coderbyte, we take a more principled approach to product management and have discovered straightforward and innovative ways to detect cheating without sacrificing the candidate experience.

For example, you can mask coding challenge titles so that candidates can’t easily search them and find solutions online. We also embed Google search capabilities right into the code editor so the candidates can have a more real-world experience that mimics how they’d code and use third-party resources in an actual job. We record their keystrokes so that hiring teams can gain a deeper understanding of how a candidate thinks, troubleshoots, and codes, all without invading privacy or introducing bias.

Automation can and should enable our industry to be more inclusive and meritocratic. If you follow these best practices, you can create effective and unbiased coding assessments and interviews that developers won’t hate. You’ll dramatically improve your hiring processes and be a model organization for the industry.