Designing a Better Hiring Process

At Simon, transparency is a foundational value. As I articulated in my post on our remote culture, we believe that sharing more knowledge with all team members — like our burn, our revenues, our growth plans, and the rationale behind choices small and large — strengthens the organization.

We extend that commitment to transparency to the recruitment process as well — if you didn’t know, we’re hiring — and we’ve spent some time outlining below how we think about bringing on a new team member. Here’s how we coach our team to think about hiring, and building a partnership with great candidates that leaves both sides feeling good, regardless of the outcome.

The interview process outlined here is adopted from processes used in some of the previous places we’ve worked, including Apple, Google, and Etsy. Our process is very much a work in progress, and we’re constantly improving how we do things.

The Warm-up

It’s important to remember that when you’re interviewing candidates, you’re having a discussion with another human being. In fact, this other human being may be quite nervous and anxious. In contrast, you’re most likely highly distracted by the current pull request you’re currently hacking on.

The goal of the warm-up is to acclimate both the candidate as well as yourself to the conversation: set expectations for the interview, and introduce yourself.

Set expectations: Quickly go through the interview format. Spend time emphasizing that you’re particularly interested in getting to know them as a person: their background, their strengths, and career direction — where do they want to be in 5 or 10 years?

Introduce yourself: Take the candidate through your role in the organization. What are your strengths, what projects did you recently ship, and what do you love about your job? Also make sure to relate your strengths to the candidate’s. “It looks like we both have experience in D3”, or “It looks like you’ve run Spark in production — I’d love to know more.”

In general, try to get the candidate excited about the role they’re interviewing for, and don’t forget that if you end up extending an offer, you still need to convince them to accept it.

Design Decisions

Ask the candidate about their background and try to get a sense of how they “designed” the path that comprises their career. Dig deep into inflection points in their career and understand what motivated them to make certain choices and eschew others.

For example, we recently hired a candidate who previously worked at eBay and then spent a year at a fellowship building a startup prototype as part of a venture capital fellowship program. Having him describe the circumstances around this transition helped us understand that he’s probably a good fit for our 10-ish person startup; he preferred the light-weight do anything development style during his fellowship, but also enjoyed the problem structure provided by a larger organization (with a more mature product).

It’s also critical here to make sure the conversation stays two-sided. Oftentimes when asked about their backgrounds, candidates will depart on a 15 minute soliloquy that they’ve rehearsed and recited to everyone else. Instead of asking an open ended question like, “tell me about your background”, instead say something like, “I see that you worked as an operations engineer before working on data. Why the change?”

Micro-Benchmarks

Experienced engineers know lots of stuff, and micro-benchmarks are designed to dig into specific areas of knowledge that the candidate claims as a core competency. Micro-benchmarks should be straightforward demonstrations of skill and should take no more than a minute each to complete. For qualified candidates, they should be relatively easy and for unqualified candidates, they’re fail-fast methods for determining areas of weakness.

For example, if a candidate claims to be proficient in SQL: Explain the difference between a left join and an outer join? or (harder) How are null values used in taking SQL averages?

If a candidate claims proficiency in python: How does python imports work and how does PYTHONPATH work?

Or for a data scientist candidate, ask some specific questions around parameter tuning for popular machine learning algorithms.

Code Review

As a second component in evaluating raw technical skills, we ask all candidates to do some coding with us during the interview. While an interview is far from any real-world setting in which one would actually sit down and write software, we still maintain it to be a critical aspect of the process.

Our ideal coding exercise has four components:

Interactive: One of the primary goals of the exercise is to get a sense of what it’s like to work with someone. For phone screens, we use Etherpad as a way of writing code interactively in a simple editing environment. We’re a Python shop, and some of us also use iPython notebooks as another form of shared coding environment.

Underspecify: Make sure the problem isn’t perfectly specified. As the candidate asks questions, let them guide you in how to handle various corner cases, etc.

Start easy: Interview coding is stressful, so if possible start with a simple instantiation of your problem that’s relatively easy to complete.

Verify & test: When completed, have the candidate verify their results and walk through examples. Also ask them how they’d go about testing their code.

Performance Test

For us, the most important aspect of any candidate’s performance is their ability to ship code. There are lots of super smart engineers out there who have high technical proficiency, can write code with you all day long, have super impressive resumes, yet can’t claim ownership for any code running in production.

An engineer who can consistently ship code must be able to (1) possess a degree of technical proficiency to integrate their software into a larger context, (2) effectively work with others and incorporate their critiques and concerns, (3) take a concept from end-to-end and build something between “good enough” but not “perfect” in a timely fashion, (4) understand and take ownership of operational challenges and burdens that come with maintaining deployed production systems, and (5) numerous other intangibles.

Given this, using principled methods in assessing one’s ability to ship code is hard. But, on the other hand, if it walks like a duck…

Go through the candidate’s experience and ask specific questions about code ownership and project ownership. If they spent a year working on a team that overhauled search functionality, where exactly did they contribute, which components did they build, what were some of the challenges they faced during development, and what if any operational problems arose during deployment? A core contributor will be able to speak very directly and specifically to each of these points, which should give you great comfort that this individual will be a core contributor to your team as well.

We’re also big fans of side projects as they’re great indicators of code shipping abilities.

Conclusions

Effective hiring is part art and part science. Determining individual character, motivation, and company fit requires a degree of subjectivity that’s hard to prescribe. While assessing soft skills is very much an art, assessing hard technical skills is much more of a science. And since the cornerstone of good science is repeatability, we’ve developed the above interview process for use with all engineering candidates.

As we scale, it remains to be seen how much we deviate as a whole from the relatively prescriptive process outlined here. Many senior engineers have strong opinions on how to interview and come with their own preferred format. Our view to date has been to prefer a unified approach over one that may be comfortable to certain individuals.

Either way, this is where the science of hiring meets the art and is certainly a topic we’ll explore more future posts.