Hiring Decisions: Motive, Method, Data, and Debrief

It’s Christmas and I know that money is tight for many. I’ve put this article together to help anyone that is looking for a new job and wants to understand how people make hiring decisions. I’ve been blogging quite a bit about Docker recently so this will be a nice break. If you would like to learn about Docker my new book Docker in Action will be half off on the 26th with promo code: dotd122614au. Merry Christmas!

Hiring will start picking up again in a few weeks once people get back from vacation. Companies will know what their budgets look like in the next year. Recruiters will be dusting off their LinkedIn accounts. If you’re looking for a job in the tech industry you should read this article. Knowing how interviewers and hiring managers think is at least as important as knowing your craft. What follows is a reflection on the many interviews and debriefs that I conducted at Amazon.com. I’m not an Amazon employee and what follows are my personal thoughts and opinions.

Amazon.com is known for having a relentlessly high hiring bar. Their interviews are technical and unforgiving. An applicant is tried for at least 4, but often 5 or 6 hours straight. Questions come in four basic varieties and are testing for either foundational computer science knowledge or leadership qualities. I’m not going to pretend that interviewing at Amazon is exactly like all of the places that you might apply. Certainly an interview for a startup is likely to stray from what I’ve learned. But I think the core ideas are the same.

Hiring Forces

I’ve never seen a case at Amazon where someone was hired for the sole purpose of filling head count. I’ve heard of that happening at other companies, and I can only imagine what happens in government offices. When people want to hire another team member it is because they need help. This is especially the case at the beginning of the year; immediately after they set direction and goals. People need to fill their teams as quickly as possible, but speed is not the only thing that matters. It is even more important not to make a bad hiring decision.

It is difficult to fire someone legally and without risking your company’s assets in court. While big companies staff legal and human resources departments to make sure they are doing things by the book, the best way to deal with the problem is not to make a bad hire in the first place.

If you’ve managed to piece together a well functioning group introducing another unknown member is a huge risk. Hiring the wrong person is poison for a team.

Preparation

Preparation is a way to pay down the debt we all acquire by being idle. Working a job that is less demanding on your skill set or neglects some component of your skill set will weaken it. It takes a ton of work to polish it again, but it is worth it in the end. Try to stay positive during this process.

If you ask yourself questions like, “When would I ever use this?” then be sure to follow through and answer that question too. Wonder about the value of that linked list question you were studying? All of these things have real world applications, see Intrusive Lists in Doom 3. I once had a person rant for the first 10 minutes of a 45 minute phone screen on how they’ve never once in 20 years used a linked list. I’m not sure what they expected that tactic to do for them. They had under prepared.

Prepare for an interview as if you want the job. If you’ve adequately prepared and you don’t get an offer, you’ll know that you did everything that you could. That either you were not a good fit or luck was against you that day. Luck is a factor that interviewers want to eliminate as much as the people applying. But not getting a job because you’re not a good fit is a good thing. You don’t want to work somewhere that you can’t. It is a very negative experience for everyone and a huge waste of time.

So your preparation efforts should match your desire for the position.

The Interview

In my interviewing experience there are two types of technical interviews. Those that test your familiarity with a specific technology, and those that ask you to solve problems while demonstrating some basic understanding of computer science.

I hate technology quiz interviews. I hate being asked those type of questions as much as I hate asking them. They are either pure memorization or answerable through reference documentation. These are quite common in interviews at smaller companies or companies outside of the tech sector. They are symptomatic of short term thinking and inexperienced interviewers. An applicant really cannot prepare for these type of questions. Either they have committed to memory their full experience with a technology or they will falter. When this happens I’ve seen interviewers start to backtrack on the value of the question or accept wildly incomplete answers. At this point the interviewer may as well just roll a die. They’ve failed to contribute valuable data to the hiring process.

Thankfully Amazon doesn’t use these. They are rare in big software organizations (I’ve heard a few horror stories from Google applicants — I’m sure those were an exception). These companies tend to favor testing an understanding of computer science fundamentals.

This type of interview is not about memorization. It is about loose familiarity with basic tools and problem solving. While there is a memorization component, these concepts crosscut all technologies. Any candidate is both more likely to have experience with this material and be able to apply that knowledge as technologies shift. Computer science fundamentals are the lowest common denominator.

Facing a technical interview can feel like standing at the business end of a shotgun. You know the questions are coming. You know that they will be new to you. Interviewing is stressful. Interviewers know this and generally won’t hold your brow or armpit sweat against you. After all, we went through the same process. We aren’t here to torture you.

Interviewers have asked their questions several times before. This is important because it helps establish a bar for the question and help the interviewer anticipate common mistakes and high quality hints. This is helpful for you to understand because you are being held to the same bar as everyone else. You are not being singled out. In truth, we want to hire each of you. When a person seems like a good fit, but is disappointing technically it can be gut wrenching.

The purpose of the interview is two-fold. The interviewer is there to try to get you to provide as much evidence as possible that you are familiar with key concepts. You are there to give as much evidence supporting a hire as possible. You are your own marketing department. After the interviewer’s questions have been satisfied, sell yourself. If there is some relevant thing that the interviewer is not asking you about, offer it. We are all there to learn as much about you as possible with very limited time constraints.

Instead of thinking about the interview as a test, think of it more like a brainstorming session. Pretend that the interviewer is there working with you to solve the questions. Ask your own questions. Think out loud. By doing so you not only focus more on the question, but also show the rest of the people in the room what it is like to work with you.

The questions will vary in their difficulty, but all are possible with proper preparation.

A quick aside: If you are an interviewer and ask “tricky” puzzle questions or some super nuanced bullshit, I hate you. You devalue the whole process. Your crap might exclude some huge number of candidates, but it doesn’t raise the hiring bar. It puts the bar on a different rack where nobody is competing. Excluding people based on a shit question doesn’t make you smart, it makes you a jerk. Stop wasting everyone’s time.

The Data

Interviewers are looking for proficiency in some variation of four competencies. Those competencies are data structures, algorithms, software design, and problem solving. At Amazon each interviewer is also looking for a specific leadership principle or set of principles. I don’t have any illusion that this is exactly the same everywhere. I’m sure Amazon is more formal than most companies when it comes to evaluating leadership and culture fit.

Interviewers evaluate responses with the following in mind:

• Were questions answered correctly?
• How long did it take to get to an answer?
• Was there any variation from the expected answer?
• Was the variation an optimization?
• Was the candidate able to deal with ambiguity? (Did they ask questions?)
• Were hints given? Were those hints out of the ordinary for other successful candidates?
• Was there some problem solving method? Was the problem solving methodology sound? (or did they just come up with an answer out of the blue?)
• Was the candidate someone that you want to work with or were they a jerk? (culture fit — surprisingly some people really are jerks in interviews)

Internal references are a huge benefit. Rules about referral bonuses might exclude that input so be careful about your priorities. If there is someone that you have worked with and would like to work with again you might not want to create a conflict of interest by taking financial benefit from their hiring. Instead you might forego that perk and participate in the debrief. I love personal referrals. There was one case in particular where the interviewers had doubts about a person’s ability to work quickly. Apparently in the interview he was a bit slow in picking up the problems, but completed them correctly and in the time allotted. They wanted to bring him in at an entry level. Having no financial ties to the decision, I offered my experience in working with him. Even today, I’ve never seen someone pick up technology or work as quickly as this guy. I’d have him on my team in a heart beat. If speed was their only concern I could put that to rest. With that contribution they made an offer at a higher level. He accepted and was quickly revered as an excellent hire.

Other data that is occasionally taken into consideration is a judgement on velocity. Companies like Amazon want to hire people that can be hired or promoted into career positions. The data used in this analysis is years of experience, rough leveling in recent job history, and education. This is tricky and in my opinion the least consistent factor. I cannot stress enough that you should not misrepresent your prior experience. Even if they don’t find out, it could end up hurting your chances.

Velocity and Leveling

For a moment pretend to be an interviewer. Consider a person who during an interview is able to answer the questions but with hints and taking some time. Maybe they stumble on topics that come with more experience like software design. To the interviewer, they seem like a great entry level candidate. But the candidate’s resume includes 5 years experience and a BS in computer science. At this point in a career you might expect that person to be experienced enough to perform at a higher level. Again, this is tricky.

At this point you know that the conversation is going to be a long one. We know that not all experience is equal and that the stress of an interview can impact performance. But do you really want to hire someone who’s demonstrated skill set is lagging their experience? On one hand you might consider offering them a job at the lower level on the other you might just pass altogether.

In making the offer you are taking two risks. First, you risk offending them. Candidates are people and customers. Offending people is antisocial and alienating customers is bad for business. Second, you risk hiring someone into a non-career position that they cannot work out of. This will eventually result in a “manage out” situation if the person does not leave on their own accord. In the mean time they can be a real energy sink on their team.

So why don’t we just level based on experience? The answer is simple. People develop an intuition for the appropriate levels of their peers. If you over hire someone relative to their skill set, you alienate their peers and the people at lower levels. People love to see their peers grow. People hate to see their peers compensated beyond their demonstrated value.

Now consider an applicant with 10 years experience. They have a MS in computer science (10 years ago) and have been working at a software company. During their interview they fail to demonstrate the ability to write even the most basic solutions in code. In questioning you discover that they’ve been lead on a mature product for 5 years. In this case there is a clear experience / ability mismatch. You cannot offer such a person a level that would be appropriate for their experience. Honestly, these are the most aggravating cases. These people are clearly capable but feel entitled by their experience rather than challenged by it. They under prepare. If you are interested in relaxing then retire or find a position where relaxing is part of the job.

Expectations should increase not relax with a candidate’s experience. Doing otherwise is a recipe for a crappy work environment and demotivates more valuable personnel.

I’ve talked about leveling quite a bit in this article, but it is not always about level or title. Even if you are in a totally flat organization you’re still likely to have varied compensation. Companies like Valve have taken this to the extreme where the company is flat and individual compensation is public knowledge. In the end it all seems the same to me.

Leveling decisions are based on demonstrated abilities not prior experience.

The Debrief and Data

Interview debriefs happen after each interviewer has finished and collected their thoughts on the experience. Sometimes interviewers share notes between interviews; tipping the next interviewer to probe into something specific. At Amazon this was rare if not specifically disallowed. Each interviewer was tasked to gather data on the person’s knowledge of a specific competency.

Once all of the data is gathered interviewers vote on the candidate. If there is any split, the debrief leader will ask each interviewer for a brief narrative that drove their decision. This is the real action.

Hopefully the candidate has provided at least a few interviewers with enough ammo to defend them at this stage. As each interviewer speaks the rest will listen and ask questions. If the interviewer has the data to do so, they will defend their vote to make an offer. If not, they will present what was found lacking. After the data has been presented the leader will ask for another vote, or ask specific interviewers if their vote has changed.

It is rare for debriefs to go smoothly. I’ve seen a few nearly turn into wrestling matches. The stronger that you can make your interviewers the better. In the debrief the only thing that matters is the data. The more contentious the interview, the more thoroughly the data is examined and the more likely a leveling conversation will take place.

Four Key Points

• Understand the criteria for your evaluation.
  Most of the time your recruiter will provide you with a guide to your evaluation criteria. Read it. All of it. Read it again. Read it like you want the job. If you don’t take the time to read Amazon’s leadership principles or the Netflix culture slide deck how can you claim to be prepared for those interviews?
• Prepare for the interview as if you want the job.
  This is a big one. The questions you will face are not easy. But their solutions are not secret either. Others have been asked and answered the same questions. Practice more than you think you should. If you decide not to study trees or recursive solutions how can you claim to be ready for an interview?
• Understand your own narrative.
  Interviewers expect that you will be able to teach them about something that you’ve done in the past. Don’t belittle a small project or something you don’t feel would impress. Just teach. This is a great way to demonstrate overall proficiency. If you don’t understand what you’ve done, how can you expect an interviewer to do so?
• Give your interviewers ammo for your defense.
  We want to hire you. We need something to use to defend that desire. We need you to show us that you’ve earned it so that we can turn around and defend you in the debrief. An interviewer is your best friend. If you can’t defend yourself, how can you expect them to?