At Paxos, we give all software engineering candidates a short take-home test before deciding if we want to move forward with an in-person interview. We’ve found it works really well, so we want to share our process and why we think it’s so effective.
Our hiring process in one easy flow: phone interview, take-home test, in-person interview, and offer!
We don’t just give out take-home tests for our own benefit; it saves candidates time as well. Our offer rate after in-person interviews is much higher than if we gave interviews to everyone who passed a basic phone screen; we’re often able to confidently give candidates a yes-or-no answer after only a few hours of in-person interviews. The take-home test eliminates the need to do marathon 6-hour interview days or multiple rounds of interviews.
The biggest advantage of the take-home test, however, is that candidates have the opportunity to showcase their best work under more realistic working scenarios. Our take-home test can be completed on the candidate’s schedule, which is especially useful if their full-time job keeps them busy during work hours. More importantly, we know that writing code live can be nerve-wracking, and it can be hard to do a great job under pressure. If someone aces an in-person code test they’re probably very talented, but conversely if they do a mediocre job, does that mean their skills are mediocre? Perhaps the question was explained in a confusing way, the candidate was thrown off by working on a foreign development environment, exhausted from a day of interviews, tired from a bad night of sleep, freezes on writing code on a whiteboard. The list goes on.
In a competitive market for software engineers, we can’t afford to only hire candidates with perfect in-person interviews. Human communication is a lossy protocol!
When we grade a submission, we’re evaluating it like we would a pull request on our codebase; we expect it to be great! We always make it clear to candidates that we want them to show off their best work. Two engineers grade each submission to ensure we are both thorough and fair. Here are some common things that tend to cause poor code reviews:
IS YOUR CODE PERFORMANT?
Every question we ask has ideal, acceptable and bad performance. One benefit of the take-home is that you have plenty of time to think about performance and revise your work. There’s no excuse for an O(n2) answer to an O(n) problem. Bonus points go to a candidate who explains the performance of their code and how they could improve it if needed.
DID YOU GIVE YOURSELF A CODE REVIEW BEFORE YOU SUBMITTED YOUR CODE?
Since code is generally read many more times than it is written, we expect all code at Paxos to be self-reviewed before submission. Often times, candidates try to solve a problem one way and then change their approach along the way. That’s totally normal, but it’s important to go back and clean up the code so that it makes the most sense given how you ultimately solved the problem. Leaving in unused variables, commented out code and print statements for debugging demonstrates a lack of caring for your craft.
DID YOU USE THE RIGHT TOOL FOR EACH PROBLEM?
There’s often multiple correct ways to do something, but you should pick your tools carefully and then use them in the way that best demonstrates their utility.
DOES YOUR CODE FLOW LOGICALLY?
This one is hard to define, but you know it when you see it. Generally, shorter code is better code, but this heuristic has a limit. We sometimes get submissions that contain 100 files to solve a problem that can be solved in 30 lines of code (the former may be acceptable if well documented).
ARE VARIABLES AND METHODS WELL NAMED?
It’s one of the three hard problems in computer science.
DID YOU INCLUDE CLEAR AND CONCISE DOCUMENTATION?
We accept submissions in any language, so wherever there is ambiguity you should be clear about versions, dependencies, platforms and other configuration instructions.
DID YOU INCLUDE A WORKING EXAMPLE AND/OR TEST COVERAGE?
If your submission is perfect, this may not be strictly necessary. But if you made any mistakes, it’s easy to forgive a minor error if you can easily see what the candidate was trying to do.
IS YOUR USE OF COMMENTING APPROPRIATE?
Unclear parts of your code should be well commented (or better yet, made clear enough so that they don’t need comments), and obvious parts of your code should not be over-commented.
DID YOU FOLLOW THE INSTRUCTIONS?
Each question we ask includes example inputs/outputs. It’s amazing how many submissions we get that don’t meet the spec.
Sometimes candidates do an excellent job on their take-home, but when they come in for an in-person interview they’re unable to solve a simple fizzbuzz problem. To combat cases of cheating (most commonly, getting help from friends), we do two things:
- Before flying candidates in from other cities, we ask them to solve a simple live-coding question on a Google Hangout. This won’t tell us that they’re extremely talented, but it guarantees a minimal ability level.
- During their interview, we ask candidates about their code. If they’re unclear how it works, it’s a major red flag. We sometimes ask them to improve their code on the spot or to build on it to solve other problems. We think it’s fair game to ask more challenging in-depth questions since the candidate has previously had time to think through the problem.
We care about many things when evaluating a candidate, but the most important thing we’re testing in the interview process is work product. There is no better approximation than doing a test that simulates real work, especially when it’s on your own time and in the comfort of your own space. Our problems are designed to simulate the types of challenges you’d be working on at Paxos so we can see how you would think through them.
The strongest objection we get from candidates is that they’re too busy at work and don’t have time for this. Our take-home is short, so that shouldn’t hold you back. We currently ask three questions, each are solved on average in 50 lines of code (some can be done in much fewer!). They’re designed to be fun problems, so if you enjoy writing code you should enjoy the challenge. We’ve been iterating for 9 months on our choice of questions, wording and examples, so we think it’s pretty good.
Though take-home tests aren’t a silver bullet, they’ve been really helpful for us at getting a fuller picture of a candidate’s abilities while saving everyone time in the process. Candidates seem to enjoy the questions, too!
Originally published at www.paxos.com on May 11, 2017.
