find_open_restaurants: An engineering take-home test dissected

There has been some great discussion in my Twitter stream this week about the “take-home test” — the engineering recruiting practice of having a candidate work on an engineering project, usually as a precursor to on-site interviews.

I wanted to give my take on take-homes, and thought a good way to illustrate it would be to break apart the test I created which we used to give candidates at Sosh.

The exercise

Hi Candidate,

Please find below our coding exercise. You can use whatever programming language you feel most comfortable in (replace “Python datetime” with whatever the native object is in the language to express a Date & Time).

Given the attached CSV data file, write a function find_open_restaurants(csv_filename, search_datetime) which takes as parameters a filename and a Python datetime object and returns a list of restaurant names which are open on that date and time. Optimized solutions are nice, but correct solutions are more important!

Assumptions:

* If a day of the week is not listed, the restaurant is closed on that day

* All times are local — don’t worry about timezone-awareness

* The CSV file will be well-formed

If you have any questions, let me know. Submit your solution as a ZIP file at the link below.

Rod.

What I got right

• It is understandable. One sentence describes the problem, and it’s real-world enough that anyone should be able to grasp what’s being asked for.
• It’s based on real work. This isn’t some bullshit trick question where the answer is “use two pointers”. The question comes from real work I had to do one day, writing a script to ingest opening hours from a scraped data set. It was typical of a task an engineer might be asked to do at Sosh.
• It’s harder than it looks. Ah, dates and times. Anyone who’s had to work with them is scarred. What could seem simple at first glance (Naïve approach: If the restaurant is open on the day, see if the requested time is greater than the opening timeand less than the closing time) breaks when you have restaurants open past midnight. Nothing in the question hints at this, so it was only discoverable by running your code and realizing that some restaurants you expected to see didn’t get returned. This is a question that rewards attention to detail.
• There’s more than one way to do it. Part of the delight in reviewing submissions was the variety of ways people cracked it. I saw all manner of creative solutions to the parsing of the strings, the data structures, the calculations. Some were complex and wrong, some were simple and perfect. Some came with unit tests, some didn’t even run. I got to learn a lot about the way the candidate approached coding and the tools in their belt.
• Candidates could work in their comfort zone. They were using their personal computers and the language of their choice, so they had a familiar development environment, without someone looking over their shoulder. Hopefully this was the lowest-stress way to test someone’s coding skills.
• It led into the in-person interview. If the code submission had a bug in it (spoiler warning: they usually did!), the candidate’s in-person technical test was to debug it with me. I’d point out a case that failed and we’d spend time fixing it together. The ability to debug code you wrote a week ago and haven’t thought about since is a vital skill for an engineer!

What I got wrong

• We gave the test to too many candidates. For a while, every candidate for Sosh got this exercise. Server, web front-end, data, iOS, junior, senior, whatever — you got some variant of this. It was created at a time when the team was small, and we were all generalists. While it was a good test of a certain mindset, we should have been faster to adapt to the different skill sets we needed and create better tests.
• It took too long to complete. We didn’t track how long candidates were spending on it, but I suspect that a good solution probably took over four hours, and could definitely balloon past that. That’s a lot of someone’s free time to take up.
• It looked too simple. Challenging questions are important — you want an engineer to feel a bit stretched in your interview process. If they feel like it was all easy, you’ll have difficulty closing them because they worry they won’t learn anything at your company. Although this question was hard, the devil was in the details, so at first glance someone might think it was easy and a) not budget enough time to work on it, or b) lose interest in Sosh because this question looks boring.
• It was probably too hard. I wrote the question thinking it was simple to code, but complex to think through. But there was probably too much trivia to know before you could even really get started: how to parse a CSV file, how to break up the text strings, how to deal with datetime objects. All stuff that was easy to me because I was immersed in these problems.

Why I stopped using the exercise

I’d love to pretend I figured out all the things that were wrong and quickly changed direction. No, those are all realizations from a year or two down the road.

The real reason was: we were desperately trying to hire, and we were losing candidates when we sent them this test. Our recruiting was all outbound: emailing engineers who already had jobs and trying to convince them to come in and meet with us. They were employed so why should they spend time jumping through our hoops?

To optimize for speed, we switched to a simple code question done over the phone with a shared codepad. Just prove you were halfway capable at writing some code that ran, and perhaps for brownie points optimize it so it ran quickly. (The optimal solution? Use two pointers. Oh, the irony.) Pass that and you could join us in the office for in-person interviews with technical questions that were designed for different levels and roles. I’ll describe one of those questions in a future post.

What I would do differently

I still think there are advantages to take-home tests. What I would change about this question:

• I’d make it optional. Perhaps if you do the test, you can skip a couple of hours of in-person interviewing, so candidates are just choosing where to spend their time.
• I’d have different tests for different roles. We did a good job of this for in-person technical exercises at Sosh, and we should have extended that to the take home test.
• I’d provide more scaffolding. Maybe supply the candidate with a simple version that did the CSV parsing. Maybe unit tests, some of which passed, some which failed, and ask them to fix it.
• I’d gather data on time taken. I’d make it clear that hours spent wasn’t an evaluation criteria, but understanding the time would push us to adjust the questions as necessary.

Further reading

• Camille Fourier’s “Thoughts on Take Home Interviews”
• Julia Grace’s “A Walkthrough Guide to Finding an Engineering Job at Slack”
• “How to Hire for a Startup” (via Maia Bittner)