Quantitative Hiring for Fun and Profit

Let’s talk about your interviewing process. Specifically, how do you evaluate candidates? Intuition? Opinion? Experience? What if I told you there is a way to evaluate your candidates that is consistent, fair, and makes it really easy to justify your hiring decisions? It’s simple but a little more time consuming. In exchange for the extra time investment, you will ensure that everyone on your hiring team is operating from the same set of assumptions about how to evaluate candidates. You’ll take candidate evaluation beyond simplistic scorecards, while allowing the hiring team to fully own the evaluation process. I’d argue that adopting this hiring process will add value to your organization because your enhanced decision making superpower will do a better job of matching candidates to positions. You’ll find the best candidate for the position.Because your new hires will be better fits for both the position and your culture, they will perform better, and maybe even stick around longer. I’m about to describe a way to quantify the qualitative. You’ll learn how to pair rank and weight your feelings. It’s awesome.

There’s two parts to this thing. Part one is your criteria. Part two is the actual candidate evaluation. This whole process takes a little more time than just deciding on which behaviorally based questions you’re going to use, or whatever. You’re gonna need your team. One way or another you’re going to need to figure out who will be participating in the the hiring process, make sure everyone is familiar with the job description, and get them all in a room together for about an hour to sort out the criteria.

PART ONE

Once you’ve got everyone in the room, you’ll lead the team in a brainstorming session around what they believe are the most important qualities embodied by the new hire. Take 10 minutes or so, and capture everyone’s thoughts. A whiteboard will be helpful here. Now spend a few minutes consolidating the ideas, ie, “experienced developer” and “seasoned performer” can be combined into one phrase. Aim for no more than 10–12 descriptive words or phrases. Congratulations, you just finished the qualitative portion of this exercise.

Here comes the quantitative part. Ask the team to look at the first descriptive word or phrase (D1) and compare it to the second descriptive word or phrase (D2), and ask them, if they could only choose one the descriptive qualities, which would it be? This can feel like an impossible question sometimes, but the team needs to imagine two candidates, and each embodies only one of the qualities. Which imaginary candidate would be most successful in the position? That’s the “winner”.

Now let’s pause here for a moment to consider two ways of doing this small comparative exercise. The first way is straight majority rules — ask everyone to vote for one or the other, and go with the one that gets the most votes. This method is quickest. The second way is to tell everyone that you can’t proceed until they all agree on D1 vs D2. This method is slower, but more inclusive. It also takes some group facilitation skills. I’ve done both, and I can say that in larger groups, you don’t gain much for going slow. Smaller groups can come to consensus faster, and generally buy into the overall process more deeply when made to hash out their opinions, but if time is precious, just vote.

What you’ve just done is called “pair ranking”, and it is how you’ll process the whole list of descriptors. D1 v D2; D1 v D3; D1 v D4, then D2 v D3; D2 v D4, and so on until you have matched every descriptive word/phrase against all the others. If you’ve done the majority voting selection method, you’ll have to count the total number of votes, and the number of votes for each descriptive word/phrase. This will allow you to rank each descriptive word/phrase (D count divided by total count). If you’ve done the consensus method, the formula is the same but you’ll be counting fewer hash marks. This is pretty much as difficult as the math gets in this exercise. When you do the division, you’ll end up with a number that can be expressed as a percentage. Write out your descriptive phrases in order of percentage, highest percentage first. Consider not belaboring every single descriptor that gets a percentage greater than zero. I typically stop at 10% and either ignore the leftover percentages, or spread them out among the “winners”. The point here is to have a shorter list of no less than three weighted descriptors. Five weighted descriptors is a great number to aim for, but remember that the more descriptors you use, the more time the evaluation process will consume.

With your weighted descriptors in hand, you now know which qualities to look for in your candidates, and the importance of each quality. Pretty cool, huh? That’s enough work for the team for one day. Make sure everyone on the hiring team has a copy of the weighted descriptors. At this point you can proceed with your usual method of developing interview questions, but try to develop questions that relate to the list of weighted descriptors. After all, these descriptors are by definition the most important qualities for success in the position, as determined by the smart people on your hiring team. It is a good idea to get this work done well in advance of actually interviewing candidates. The process of ranking and weighting descriptors helps anchor their importance in the minds of your hiring team, and will make the final evaluations of candidates much smoother.

PART TWO

Once you’ve interviewed all your candidates, reconvene the hiring team to begin to pair rank the candidates against each weighted descriptor. Start with the first candidate (C1) vs the second candidate (C2) along your most heavily weighted descriptor (D1). Which candidate better embodies D1? Tally your votes or make a hash mark by that candidate, and continue with the rankings along D1. C1 v C3; C1 v C4, etc. Keep doing the rankings until every candidate has been evaluated against every other candidate for each descriptor. Usually you can see an obvious pattern emerging about halfway through the exercise, but for the sake of completeness keep going until the last ranking. This is where your earlier decision about how many descriptors you weight comes into play. This step takes time, and if your team is unfamiliar with the process they may push back against it, preferring to stick with their intuition. Ask them to stay with you, and don’t be shy about re-explaining the overall process again. The payoff is worth it when you see the lightbulb go off in their heads and they understand what they just did.

Again, this is a good use of a whiteboard. I put descriptors on the left and candidate names across the top. Often the hashmarks can make the winner seem obvious, but usually it is a good idea to multiply hashmarks by the weight of the descriptor. So if your D1 is weighted at .5 and C1 has gathered 11 hash marks, the D1 score for C1 will be 5.5 (.5x11=5.5). Once you have all these scores calculated by descriptor and candidate, add them all up by candidate to determine the winner. Most of the time the winner will be indisputable. Occasionally two candidates will be in a horserace and the final scores will be really close. When this happens, it’s time to have some further discussions with the team to come to consensus. Regardless of whether the winner is clear or the decision needs more massaging, check in with your team for a final gut check around the decision. If you’ve been diligent about the process, by the end everyone should understand what they just did, and should be able to validate the results against their intuition.

That’s it! You’ve successfully quantified the qualitative. Sometimes you’ll have people on the hiring team who are very resistant to engaging in this process. They’ll be more trusting of the old intuition or instinct or gut feeling. Acknowledge their preference and point out that 1) their intuition or gut feeling is going to come in really useful during the pair ranking process, and 2) encourage them to track their gut feelings during the interviews and compare them to the outcome of the quantitative process. It may not always match up, but they’ll feel valued by not having their preference dismissed out of hand, and you might even make them into a convert for this method of quantifying the qualitative!

POSTSCRIPT

If you’ve never encountered anything like this, it’s a great idea to practice it on your own decisions first. You can use it with anything — buying a car, deciding on a place to go to dinner, deliberating between vendors, etc. Avoid getting caught up in building an Excel spreadsheet to do some of this thinking for you. Just use pencil and paper for your hash marks. It’s quick and easy for personal decisions. Don’t forget to gut check the final result.