Objective Interviewing is Impossible

The following thoughts are entirely mine and not those of my employer.

One of the most interesting and educational aspects of working as an engineer at a young startup is your participation in all facets of company life. I wrote a moderate amount of code in my first eighteen months at the startup but the bulk of my time was spent context switching between a variety of different roles that were arguably not part of my original job description (and perhaps one of the reasons I might make a more effective manager than an individual contributor).

During this time I did everything from writing docs to providing customer support (primarily to our community, i.e. open source users) to recruiting. Of these, recruiting took the most time, as the company grew from 21 employees (when I started) to the 180+ employees now. At the end of one quarter in 2014, Greenhouse, the recruiting management system we use (essentially Salesforce for hiring), I was ranked as having conducted the most interviews amongst all of my colleagues (including our first recruiter). All that said, the following is all highly anecdotal (but I wish I had more numbers to back it up).

Recruiting software engineers is a contentious topic, ranging from resume screening all the way to interviewing. Engineers are notoriously ineffective at conducting objective interviews, asking abstract problems that are often unrepresentative of the actual day-to-day work that will be done (re-implement a fundamental library function from scratch, just because), asking you to solve a problem in an environment that doesn’t reflect the tools most programmers have available on a day to day basis (coding into a Google Doc, ew) and finally asking questions where often the only way to do well is to have done that or a similar question before (find a cycle in this linked list plx).

Prior to joining, I interviewed around extensively, being in the privileged position of being a graduate student at a prestigious computer science department and having had positive and less positive prior work experience. I applied to 30 companies, interviewed at 15 and finally received offers from 6 of these (the others failed me, or I withdrew from the application process).

Many of my interviews were subject to the problems I described above. This was a frustrating experience as a candidate (where you feel you aren’t being set up for success, or worse, are being set up for failure) and a waste of time for the company interviewing.

I took this pain to heart and aim to conduct my interviews in what I consider a more sensible way. Lately as the company has grown, I wanted to share what I think is now important when hiring. Note that the below is my personal interview criteria which isn’t necessarily the same as what the company as a whole uses (although there is a great deal of overlap).

Purely objective interviewing (i.e. assessing a candidate on pure merit) is definitively impossible for two reasons: the first is that humans are immensely complicated and the second is that the roles we’re hiring humans for are also complicated (especially in startups where employees wear many hats). We attempt to benchmark humans using various heuristics for “merit” in a uniform way (she has X years of experience, he has programmed in Y language, she has worked at Z competitor, he can answer this technical question) but these are too simple to map well to day to day merit in the workplace.

For example, many students graduate from Cambridge with a fantastic computer science degree. This makes it significantly easier to get interviews and to pass those interviews but they often fall flat in the workplace, due to a sense of entitlement that dampens their enthusiasm for what they might consider menial work. (I’m picking on Cambridge here because of my own availability bias but from my work and recruiting experience this seems to be the case for students from most prestigious universities.) In this case, our heuristic: educational background, maps poorly to the role: entry level software development. Indeed, some of my favourite colleagues are those that studied at universities I had never heard of, or had studied degrees that tangentially related, if at all, to the job they do now.

Sometimes we look to hire against heuristics that are (almost) impossible to fulfill. For example, when first hiring interns we had the mandate to go after candidates who 1) had a great degree (preferably a Master’s or higher) and 2) had some prior work experience and 3) were close to or had already graduated. Guess what, a candidate that good is already gone: Google gave them an offer a week into their Master’s program.

There’s also this common situation that software engineers love to make fun of: “we’re looking for a candidate who has 10 years of experience in [technology that has not existed for 10 years]”. Even if the technology has, it may be niche or novel enough that there just aren’t enough of those people. It’s foolish and generally lazy to assume that accomplished candidates with a background that doesn’t match exactly is not a fit.

Given that these traditional heuristics map poorly to reality, I like to assess candidates on three higher-level principles. These represent my current thoughts, refined after screening several hundred resumes, carrying out hundreds of interviews and assessing dozens of programming challenges. What we do almost certainly isn’t perfect but it’s not bad; I really enjoy working with nearly all of the people I’ve helped hire.

(For context, our product and livelihood is primarily distributed systems oriented.)

Aptitude

The main question we’re trying to answer when we assess a candidate is whether or not they can perform the job they’re interviewing for. One popular, necessary but lazy way to do this is to consider only those candidates who have exactly the skills and experience you are hiring for. This is inadequate because often they don’t exist or because you’re actually stifling diversity and creativity by hiring people who fit a mould exactly.

Given that it’s often impossible to find perfectly qualified candidates (particularly in the young-ish field of distributed systems), you more often than not look for the next best thing: those that have the aptitude to grow into that role. A former manager of mine would always talk about the trajectory of young hires — how quickly do you think they’ll learn? I think this applies equally to experienced hires. This is really, really difficult to assess over a few short meetings but there are some obvious things you can look for:

• Have they researched your company already? This already demonstrates an eagerness to grow their knowledge (and a certain amount of diligence) that is a hugely positive sign.
• When describing a topic that they don’t know, do they ask insightful questions? What is their attitude to novel information? It’s important to consider nerves here: some candidates may be spooked by an unknown topic after interviewing under pressure to perform for many hours.
• What is their process for tackling a problem? This is the obvious intention when tasking candidates to solve abstract problems but interviewers fall into the trap of fixating on getting the candidate to the ideal solution (don’t do this!).
• There are secondary measures you can use here that are covered by the “washy” interpersonal questions that are often dumped upon, e.g. “tell me about a recent personal project you’re proud of”. Obviously it’s not always possible to do this given social and family obligations but it is nice when a candidate is sufficiently passionate about a hobby to spend their free time progressing a project.

(An internship is an ideal period of time to assess aptitude, notably when it’s hard for a candidate to point to a plethora of past work experience. We recently had a fantastic intern who came in without a degree in computer science or any prior software engineering internship experience. At the end of the summer, he was performing at a level that exceeds many full-time engineers. Aptitude: yes.)

Technical capability

Technical capability is ostensibly the easiest of the lot to assess but interviewers fail at this often. The implicit assumption is that we want to figure out if the candidate is able to program. Beyond this though, what I want to know is what the quality of their code is like and how easy it would be to maintain.

Quality encompasses so many different things that are mostly obvious:

• Are there tests?
• Is the coding style idiomatic?
• Is the code/project somewhat documented?
• Is the code performant? (Algorithmic complexity is the most obvious manifestation of this.)

More than that though is to assess how easy the code would be to maintain. Seasoned programmers know that you often spend more time reading code (while scratching your head) than writing code: particularly when working on established codebases. One of the most useful things that I learned from my colleagues at Last.fm was how to write clean, re-usable (and one might even say, beautiful) code. This is something that can be taught, especially if our candidate has had no or minimal professional software engineering experience. Often code clarity is correlated with a logical approach to solving problems.

Finally, what code should you assess? There’s two ways I like to this and one I absolutely hate (as both a candidate and an interviewer).

Code samples are an obvious start and are a great way to assess technical capability prior to the first interview. Got a GitHub profile that’s lit up green? I’ll take a look. It is extremely useful for students who have no prior work experience. One intern candidate maintained a dating website bot on GitHub. The ethics of this project aside, I was impressed that he maintained a project that had multiple contributors and was written in his free time and decided to interview him.

Assuming the code samples and a non-technical introductory call go well, how do you assess the quality of someone’s code in a more formal setting? While many companies LOVE whiteboard interviews, I am of the opinion that this is a TERRIBLE way to assess how well someone writes code. What we do, a method that we did at my previous employer, Last.fm, too and one that I infinitely prefer, is to ask the candidate to complete a take home programming exercise.

We typically provide users with a blurb which contains an example class (at Last.fm we provided a number of Java interfaces) and ask them to spend four hours on the challenge. This has a number of benefits:

• Candidates can use the tools they feel comfortable with. This is a more accurate of their actual working style than coding into a (damned) Google Doc.
• Candidates can do the test at a time and in a location that is convenient to them. If they’re using to coding on two screens, or in a dark corner, or at 2am, they can do this.
• We’re able to assess how they might treat a real project: submissions are assessed on completeness. Did they attempt to write tests? How’s their documentation?
• We’re able to develop a baseline of what we consider a “good” contribution, having received and assessed many of these challenges.

It has some disadvantages too, of course:

• 4 hours is a large chunk of time to invest in a coding exercise, which doesn’t work well for some candidates. For those with extensive open source contributions, we often waive this exercise. However, I try to justify it as a better alternative to hours of remote phone interviews. This tends to act as something of a filter.
• Very rarely, candidates haven’t had access to a personal computer with a readily available development environment and spend a long time setting it up.
• There is often some language bias (we are language agnostic and tend to pick what we consider the best tool for the job). Candidates who pick a terse programming language often do better than those that use overly verbose languages (an amusing miscommunication once had a candidate attempt to implement our backend challenge using JavaScript).

Our coding challenge is usually the next step after the introductory call, which is followed by a code review session that ideally mimics what would happen during the normal process of shipping code. This is usually followed by a couple of remote interviews and then a full day of interviews.

When we initially started doing these full day interviews, we used to ask candidates to solve a bug as part of a pair programming exercise. This allowed us to assess how easy the candidate was to work with, and often let us figure out if their challenge submission was an accurate reflection of their productivity. However, as our open source projects matured (for asking someone to work on a closed source project supposedly has some IP nuances), the codebases became complicated enough that fixing a bug within a few hours became tough.

Instead, we take a similar approach to pair programming that we do for the coding challenge. Candidates are provided with a known problem that we provide to every candidate. By design it is loosely defined (the intention being to reflect the vagueness of an actual bug report) and requires additional discovery to complete. Again, this lets us compare candidate submissions across multiple candidates in a consistent and fair way.

Code isn’t the only aspect of someone’s technical capability, which generally also extends to understanding of core technical concepts. For me, these are covered in architectural interviews, where I will either ask the candidate to describe or design distributed systems architecture or we will work through an example derived from reality.

The caveat of this sort of interview (especially with students) is that candidates may not have had the opportunity to work with these sort of systems (what’s a Kafka?) — in which case, I focus more on an example derived from a live system and walk them through the main concepts. These are concepts that I would expect nearly every computer scientist, even those that are students, to be familiar with, such as a queue, relational database and so on.

A final note here is that I don’t explicitly interview candidates on data structures and algorithms. The only people who do well on these are recent university graduates or people who have been preparing for Google-style interviews (yes, yes, sweeping assertion). A candidate’s ability to understand and use data structures (particularly those that are idiomatic) should be obvious from their coding challenge submission.

Communication skills

Communication skills are possibly the most important of these three principles since we can’t yet work in a caffeine-fed vacuum with fast internet access. As professionals, notably including those who wear many hats at a startup, it is mandatory to be able to communicate well to succeed. Primarily to your colleagues, then to your customers, to community users, to other engineers (when evangelising our software) and so on. Arguably, in not-small company, most engineers are specialized enough to only ever need to communicate well to their colleagues but these expectations expand with responsibility (i.e. a tech lead is supposed to communicate well across the organisation, and so on).

Communication as an assessment area is multi-faceted but also quite simple:

• Can you explain a technical concept that you understand clearly to someone who doesn’t?
• Do you listen?
• Are you approachable / enjoyable to work with?
• And the corollary of the previous point: are you unpleasant to work with?

One of my favourite memories of the early days of the company is that you could turn around and ask any single engineer for help with your problem, regardless of how inane it was (within reason). As someone who considers themselves relatively junior, that was wonderful and some of the early employees at Mesosphere made this my best job yet. Part of this was the obvious close knit and familial nature afforded by having so few people and the consequently uncomplicated organisational structure (i.e. none). That said, part of it was the decision to hire early employees who were passionate, knowledgeable and (most importantly) highly approachable.

As I say, this is hardly a perfect interview process. It is, in my subjective opinion, an improvement on the well intentioned but misguided hiring processes that engineers are often subjected to. The interesting question for me is to see if this can scale. One of the strengths of the Google interview process is that it is heavily quantitative in nature and this allows them to consistently set an extremely high bar. My bar is perhaps equally high but along different dimensions which are harder to measure numerically and in a consistent way.

(One area I entirely neglected in this writeup is the topic of diversity. This is something we do consider, heavily, and, like many others in the software business, continue to struggle with. That said, it’s a complicated topic and one that is best addressed directly in another article.)