The Case Against Aptitude Tests

“Sir, give me two minutes to present my case. If you’re convinced, great. Otherwise, I wouldn’t bother you again.”

Hiring is complicated. It makes us measure and compare a lot of human behaviors. The problem is, human behaviors are not easily measurable. And whenever we encounter something that is not easily measurable, we find or define something else which is a close approximation of what we actually want to measure. This new metric has to be easily measurable. For example, when we want to check a candidate’s speed and accuracy with regard to problem-solving, we can’t measure it directly. Hence, we define a close approximation of this using a set of questions about problem-solving and calculate a measure called ‘Aptitude Score’ instead. Unlike intelligence, an aptitude score is easily quantifiable and comparable.

This is not a new phenomenon. Let’s say you run a blog and want to know which one of your blog posts are most relevant to its readers. Measuring relevancy is a difficult problem. So, we define a close approximation — that if more people are clicking the post, it implies that it is relevant to more people. We approximate relevancy as open rate or click rate.

This methodology works really great. You get to know what kind of content is more relevant to your readers, you write more of that, and your business grows. You want to know which candidates in a pool are good at problem-solving, so you apply a filter on their aptitude scores, and there you have it.

However, just like every product has a life cycle, every metric has a life cycle too. Goodhart’s law is a brilliant way to describe this life cycle:

“When a measure becomes a target, it ceases to be a good measure.”

Let’s say you have a team of writers who write blog posts for you, and you tell them that you want them to write more relevant content for readers. They ask you to define ‘relevant’. You’re smart, so you tell them to approximate relevancy as open rate. It works for a while but now your writers are not working towards making more relevant content, but rather, writing content that gathers more clicks. Essentially, you just invented clickbait. As soon as open rate became a target, it ceased to be a good measure.

Coming back to hiring, the candidates no longer have to be quick and accurate at problem-solving — they just have to be good at getting good scores on their aptitude tests. You ask them “A die is rolled twice. What is the probability of getting a sum equal to 9?” They aren’t really thinking about logically reaching a solution. They are, instead, writing formulas and crunching numbers and applying the tips and tricks they learned during their CAT preparations. As soon as aptitude score became a target, it stopped measuring a candidate’s problem-solving skills.

Of course, you can still get a good aptitude score using actual problem-solving skills just like you can actually write relevant content for a blog’s readers, but there are enough people solely using shortcuts to get a better score just as there are enough writers getting better open rates through clickbait titles. Ultimately, the end result is, you see a lot of candidates in subsequent interview rounds that do not have the problem-solving skills and aptitude that you believed you had already tested. You are not shortlisting problem solvers. You are shortlisting people who read a 600 pages long book in the last 3 months.

“Sir?”

“I’m sorry I didn’t catch you. I only talk to people who can mentally calculate 4.5% of 73 in less than 6 seconds.”