Tired: Engineering Interviews; Hired: Engineering Auditions.
S̶i̶d̶e̶b̶a̶r̶:̶ ̶W̶e̶’̶r̶e̶ ̶h̶i̶r̶i̶n̶g̶.
Whenever the maligned topic of Engineering Interviews flits across my Twitter feed, I take a moment to chime in about how there’s no particular reason they need to be awful.
And people usually reply with tweets like “What? That works?”, “I’m going to steal that idea!” and “Didn’t you promise to blog about that like two years ago?”
Over my (lamentably) hair-loss inducing career I’ve been on both sides of terrible interviews: toy problems, hours-long unpaid “homework”, white board coding, and pondering just how and why a dead man is found alone in a desert holding only a broken match stick¹ .
After a series of disastrous interviews where perfectly good candidates kept washing out because they’d fail one lousy part of an interview (often conducted by a “Bar” “Raiser”) I became determined to design an interview that avoids all the various stupid dysfunctions I’d seen.
I failed.
I failed because I couldn’t find a single interview style or process that works to accurately evaluate every candidate (let alone even a small subset of candidates).
I gave up.
I gave up and started asking the interviewee to do the lion’s share of work in coming up with how I should evaluate them.
When I started at Popular Pays in 2015, I (along with my compatriot Sean Massa) brought this idea with us and since startups tend to involve participation in all aspects of a business, the team’s been able to experiment and refine this idea into what I think is a very humane and equitable process that respects the needs of both parties.
I’ve noticed that our current process shares more in common with an audition (where someone comes highly prepared for a specific task in advance) than a job interview (where someone shows up never quite knowing that lies ahead).
I’ve taken to calling this Audition Style.
For us, auditions occur in 6 stages:
Stage One: What’re You Great At?
Before we bring you in, we send an email describing our process:
Hello <name>, as I mentioned the next step in the process is collaborating on the design of your on-site interview.
We want our time together to give you the best possible opportunity for showing off your strengths. To that end we give you control over what we’ll work on when you come in. So, take some time to think about what we can do to highlight your best work.
Our stack is:
* Rails (as an API) for two services, using JSON-API as a data format.
* Ember (as a UI) for two apps, one serving content creators, one serving advertising agencies.
* A mobile app (iOS) for creators.
* A node.js system for accessing APIs of various social sites.
Your answers should be pretty personal to what your strengths are. Try to think think of a few and shoot me over what sorts of activities would best highlight why we’d be silly to pass you up!With that, I’ll try to find the right people on our end for the day and send over their information so you know who you’ll be meeting.
48 hours before you come in, we’ll give you access to any relevant repos and services for what you’ll be working on, just to give you a peek at our code (it’s pretty OK, but not all puppies and rainbows).
So take some time thinking about it and shoot me an email with strengths or any questions I can answer to help you wow us.
This allows the the candidate to avoid gaps and pitfalls in their skills and spend most of their time with us showing off their talents. Some strengths people have focused on in the last two years:
- Data Modeling a user and authentication system with a focus on security
- Finding and fixing poor DB performance
- Turning a bug report into an actionable bug fix
- Expressing a UI via routing
- TDDing a feature end to end
- A presentation and Q/A session of their largest OSS project
- Converting a Sketch mockup into HTML/CSS/JavaScript
- Refactoring a too-large model
And more. Curiously, nobody has yet asked for a toy coding challenge.
Stage Two: Who Can Evaluate That?
After the candidate emails us back, we discuss who would be the best people to work with them. Often these are people who shares skills or have deep context for the codebase the candidate might be working in.
But not always! If the candidate is strong in an area where we have no particular background we might pick the person with the least context, since the evaluation at this point is how well a candidate can lead us through the fog of war.
We usually pick two to three current teammates to work through two or three activities.
Stage Three: The Pitch
Combining what the candidate said and how it might fit with the team, we’ll pitch a few activities and what we’d be looking to learn from each. The choice of which we do is always up to the candidate and we make sure they know that these are just suggestions and we’d be happy to keep collaborating to find just the right set of activities for everyone.
We make sure the candidate has access to any relevant repositories and services 48 hours in advance to familiarize themselves with our world.
For example, the person who identified “Finding and fixing poor DB performance” as a strength had access to our main API’s repo, plus our New Relic and Skylight accounts.
We also send along information about who they’ll be meeting and working with. We’ve seen their LinkedIn, Github, Twitter, etc and it’s only fair to return the courtesy.
Stage Four: Show Me What You’ve Got
It’s the day! This is the easy part to explain. It’s like what you’ve seen in every interview, except everyone is very relaxed, there are no surprises, no gotchas, no unexpected digressions into the deep end to see how well you swim.
If you do design your interviews around that oddly antagonistic Gotcha Style, let me save you and your candidate a wasted day: it’s very easy to uncover what people are bad at. Most people are mostly bad at most things.
If someone aces your Gotcha Style interview you might be hiring by coincidence.
Stage Five: Evaluate The Individual
After the interview we gather the team to listen to tales. Who (if anyone) referred them? Who tech screened the candidate and what stuck out that earned them an on-site? What did we learn about this person’s strengths?
Occasionally someone doesn’t demonstrate their strengths well. In these cases we often ask: how can we learn more? Usually this comes with a quick followup email like:
Hey <name>,
Thanks so much for coming out and spending the day with us. The team really enjoyed our time together. We wanted to learn a bit more about your strength in <strength>. Can we chat quickly and brainstorm some ways of getting more details here?
People have sent along code from side projects, or done another remote pairing session, or offered up referrals who could speak to specific types of experiences.
Stage Six: Evaluate the New Team
“Every change to a team results in a new team.” — Me, pretending someone important said this by putting it in quotes.
Next, and most importantly, we ask: does this person’s strengths make the overall team better?
This is the source of most of our passes. It’s not that someone isn’t skilled, it’s just that we’re still a small team and trying to keep a balance of skills, interests, and backgrounds.
Sometimes a person’s biggest strengths just don’t match where we’re lacking or where we see a need in upcoming work. Typically these aren’t a “No”, more of a “Not Yet”. It’s best for us and, honestly, most fair for the candidate. They deserve a place where they can shine brightest.
In these cases I’ll usually ping colleagues who are hiring to let them know a solid candidate is searching, but that we just don’t have the need currently.
Does This Work?
Admittedly, I don’t have much data. We’re small company (but growing) and have done a vastly smaller number of interviews than the Googles of the world.
But, I do track what data we have:
- 16 people on-site in two years.
- 1 person where the position was accepted by someone else, so we had to stop the process.
- 9 offers.
- 5 accepted offers.
And, in an especially cockle-warming aside,
- 2 people who mysteriously ended up working where our team coincidentally happens to know some people.
We’ve also had nearly universal positive feedback, even from people who don’t receive offers.
What About Consistency?
There’s a growing body of evidence that structured and consistent interviews are the best predictor of future job performance. Plus, they help limit cultural and unconscious personal biases held by organizations and individuals.
A collaboratively designed, per-candidate interview is about 180º away from this approach as possible, and honestly our evaluation process is still entirely unstructured and vulnerable to these biases.
It hasn’t stopped us from making offers to women, people of color, LGBTQ individuals, and more but I can’t claim there’s any systemic protection against this disfunction.
Nor can I even guess how — or if — this scales beyond 10, 100, 1000 people. We’re not there yet, although some day we might be (h̵a̵v̵e̵ ̵I̵ ̵m̵e̵n̵t̵i̵o̵n̵e̵d̵ ̵w̵e̵’̵r̵e̵ ̵h̵i̵r̵i̵n̵g̵?̵).
¹First of all, I didn’t fly to California to participate in your third-rate Encyclopedia Brown fanfic. Plus, even the place that popularized questions like this says they just don’t work.
