The Headlight Guide to Great Engineering Take-Homes

The interview process in software development is inexact: Interviews vary greatly in content and no two are quite the same. Similar companies can have significantly different hiring processes.

It’s often infuriating and tiresome for candidates, who feel like the interview process is not about their practical knowledge but about the time they devote to studying for interviews.

I’ve been on both sides of it: I’ve done the FAANG problem-solving interviews (and been rejected mercilessly), and I’ve been on interview panels at companies ranging from tiny 5-person startups to gigantic companies like Bloomberg. At Etsy, my co-founder Jason and I realized how much time we spent on the initial screening process and decided to start Headlight, a platform for doing initial technical screens on behalf of our customers.

The take-home

One hiring process that’s near to our heart at Headlight is the take-home project. If you’re not familiar, take-homes are loosely a work sample assessment in which candidates are asked to complete small projects meant to demonstrate practical skills. They differ depending on the position, but for a full-stack engineer one example might be to complete a specific task using a provided API.

In theory, take-home projects have plenty of advantages:

• Accurate signal of ability: There’s quite a bit of industrial psychology research suggesting work sample assessments correlate closely with job performance.
• More practical and less academic: They’re meant to be reflective of day-to-day work, so candidates shouldn’t need to spend time studying for interviews (e.g. algorithm reviews).
• Immediately up-to-date: There’s no ongoing requirement to keep a Github profile up-to-date and pruned.
• Encourages more diverse hiring: By relying less on signals like brand-name employers or CS departments, companies using take homes can build more diverse teams.

Like Agile, though, take-homes have become corrupted and contentious. They’ve become meaningless free labor: Companies are unclear about the objectives, and they’ve become bloated projects that require too much of candidates and force them to spend unrealistic amounts of time to complete. As such, they’ve become biased towards candidates with the most free time. Looking for a new job is already a time-intensive process, so asking candidates to then complete 10+ hour take-homes (often for multiple companies) within the span of a couple weeks is unreasonable. It’s no wonder why they feel unfair to people with other commitments (e.g. parents), and it’s also clear why stronger candidates with opportunities don’t feel like doing them.

Like Agile, take-homes have become corrupted and contentious. But they can be redeemed.

Worse, candidates often don’t receive any feedback from these projects they spend days on other than binary feedback: “Yes we want to move forward,” or “no we don’t”. Candidates spend 10 hours on a project just to be told “no” with little other guidance, and it’s little wonder they’re not exactly eager to tackle the next one.

Fixing the process

Much like Agile, the process is redeemable. At Headlight, we’ve screened hundreds of candidates using our platform for both our customers and for our unique hiring tournaments. We think a lot about take-homes. The results speak for themselves: We create take-homes for our customers that are transparent, time-boxed (and enforced) to a few hours or less, and not a complete waste of time for the candidate.

We’ve identified a couple key guidelines that have helped us and other engineering organizations and we thought we’d share them with you:

• Set reasonable expectations
• Establish explicit and public evaluation guidelines
• Make the take-home worth it
• Don’t worry about the wrong problems

Set reasonable expectations

“Take-homes are unfair to people without a ton of free time, like working parents.”

That’s because you‘re asking them to spend 10 hours on a take-home! We’ve talked to a lot of candidates and heard so many horror stories of obscenely long take-home projects. One software engineer who was interviewing recently told us they were given a take-home project (from a fairly well-respected company) with an estimated time of 20 hours. The shortest take-home they received took about 8 hours. In contrast, the average Headlight take-home is 90 minutes long (and the time limit is enforced). There’s absolutely no reason a take-home project needs to be such a monumental task. Decide what specific things you’re going to evaluate candidates on, and eliminate everything not necessary to accomplish that.

Here’s a simple and common one: Developers rarely create new projects on the job. You probably don’t evaluate a potential hire on whether they can generate templated code. Provide some boilerplate. If you want to evaluate their troubleshooting and problem-solving ability, make it an explicit part of the assessment and not an incidental one. Establish a time budget, and count everything against it. You’ll find that the 10 minutes you’re asking them to spend generating boilerplate code is almost certainly better spent doing something else.

If you’re not doing the take-home yourself (or having someone you trust do it), how can you ever know what’s a reasonably good submission?

Most of these low signal tasks you’re asking of candidates will get sussed out when you do the take-home yourself — which you absolutely should be doing. If you’re not doing the take-home yourself (or having someone you trust do it), how can you ever know what’s a reasonably good submission?

Before we use a new take-home on our customers’ candidates, we always have someone complete it within a time limit and have someone else grade the submission. If there’s a mismatch about the perceived quality of the submission between the submitter and the evaluator, that’s something worth reflecting on. Either the evaluation criteria is too severe, the time limit isn’t enough, or some other combination of factors. Investigate and iterate until everyone roughly agrees it’s well-scoped and fair.

Most importantly, if you’re not going to use a platform like Headlight: Stick to the time limit when you’re developing and testing the take-home. In the absence of an enforced time limit, some candidates will spend far more than the recommended time. This then skews your idea of a quality submission — you start comparing everyone to the excellent submission you received that someone spent 20 hours on. Don’t give in to the Red Queen: You should only compare candidate submissions to your own time-boxed one, not the one someone spent an entire weekend on.

Establish explicit and public evaluation guidelines

“They take too long to grade. We don’t get a good signal from them.”

One of my biggest frustrations with take-homes at previous positions was a lack of clarity around what constituted a passing submission other than a vague gut feeling. We’d give candidates a take-home with little guidance, and then rail them later for not reading our minds.

For example, we would sometimes ask candidates to make a small client-side web application; small enough that it could easily be done with vanilla JavaScript. Right around the time React started gaining prominence quickly, candidates would sometimes include React in their submissions — and we’d trash them for it. Some people (mea culpa: including myself) would leave comments on the evaluations like “Why are they importing React for this trivial web app? It shows a lack of judgment about when to reach for a framework.”

Of course they used React, though! React is popular and we didn’t tell them what we were evaluating them on. A well-managed project will always include the terminal states, often stated as “What does this look like when it’s done?” So why do we insist that candidates guess at the terminal state here?

We really believe in sharing the evaluation guidelines at a high-level with the candidate. Tell them what they’re going to be evaluated on.

We really believe in sharing the evaluation guidelines at a high-level with the candidate. Tell them what they’re going to be evaluated on. One sample criterion might look like this for a take-home that involves interacting with an API:

We explain what “thoroughness” means in the context of this assessment (since it’s vague), and we provide an example of what we’re looking for to the candidate. We provide more detail to the evaluators about how to evaluate this category and to avoid trivializing the assessment. Notice we’re also showing what we don’t care about through omission: “Unit tests” aren’t listed or implied anywhere, even though one reasonable interpretation of “thoroughness” is “test coverage”.
This makes evaluations significantly less arbitrary. No more decisions based on “I just don’t like something about it”.

Make the take-home worth it

“Experienced candidates won’t do them. It reduces our hiring pool.”

Part of the reason candidates dislike take-homes as they exist today is that it’s a lot of work that has little upside for them: They don’t get paid for it (maybe they should be), it’s not reusable, and in general it just feels like a waste of time.

It doesn’t have to be that way. Make it part of the on-site interview:

• Ask them to explain why they chose the approach they did.
• Ask them about any tradeoffs they made.
• Ask them how they’d extend it further to add a specific feature.
• Ask them what they would’ve done with more time.

Turn it into an in-person code review and skip something else. Whiteboard interviews are stressful. Most candidates would much rather discuss a piece of code they wrote recently than find the longest palindrome in a string. Senior candidates won’t begrudge a 2-hour assessment if it avoids a 1-hour whiteboard interview.

One other thing that we’re starting to do at Headlight is to share evaluation feedback with the candidates (with the customer’s permission, of course). Candidates shouldn’t spend multiple hours on something just to hear “yes” or “no”. Consider whether providing feedback is something that you can do. If candidates can take something away from it, it’s not a complete waste of time.

Don’t worry about the wrong problems

“Can’t candidates just cheat on them?”

Quite a few companies we’ve talked to are worried about candidates cheating on take-homes. Software has sprung up in response that installs spyware on your computer to monitor your keystrokes, or watches you on a webcam during the assessment (yeah, I wouldn’t do that take-home either).

This is a pretty overblown fear: There’s very little evidence to suggest this is anything other than a rare occurrence, and a really determined cheater is going to find inventive ways around whatever process you put in front of them. If you make discussing the take-home part of the on-site interview, a trained interviewer will quickly note that the candidate doesn’t seem to understand their own code (I have seen that happen once before and yes, it did result in terminating the interview). There are places where cheating happens and it’s worth worrying about; this isn’t really one of them.

One reasonable response to all of this is, “Why the hell would I do take-homes after reading this? It seems like a ton of work.” I’ll remind you of two things:

1. The work is mostly front-loaded
2. Work samples are much more highly correlated with job performance

To the first point: Once you setup the take-home and the grading criteria, the evaluation process itself is simple. It requires regular reflection and iteration (like any other hiring process), but evaluating candidates becomes a 20-minute affair if you’ve established thorough evaluation guidelines — compare that to a phone screen that likely takes 90 minutes at least (including evaluation time).

If you could spend 10 hours to reduce the time you spent hiring by 50 hours, wouldn’t that be a huge win?

To the second point: Do some math for me right now. For the last position you hired for, how many people did you interview? How much time, in total, did you spend interviewing candidates that didn’t pan out? It’s not a stretch to say that denying a candidate after an on-site interview costs your company 10 hours (if not more) of people-hours. If you could spend 10 hours to reduce the time you spent hiring by 50 hours (assuming 10 candidates), wouldn’t that be a huge win?

You may still decide take-homes aren’t for you, and that’s fine: The goal here is mostly to get you thinking about your interview process. Interviewing is a process like any other, and it won’t be great from day one. Work on it, revise it, and be thinking about it constantly. Even if you decide take-home projects aren’t going to be part of your hiring process, take some time to think about how you can improve the process for both yourself and for candidates.

Originally published at blog.gitprime.com on August 7, 2018.