Why automated recruitment is flawed — And why both companies and candidates are losing something
We’re now living the automation boom. There’s a huge excitement about self-driving cars and the undeniable success these machines are doing not only for the companies that are developing them, but also for the majority of the population who suffers the consequences of bad drivers who create more stress and pollution than it’s necessary, and the worse: killing thousands every year. Of course, this automation, like any other, will kill countless jobs that we still don’t know how to replace, especially because now automation seems to be accelerating its pace faster than the creation of new kinds of jobs. But this text is not about the consequences of automation. It’s about the effectiveness of the process itself, specifically the automation of the recruitment process in the software development field.
No one can deny how hard and tricky is the process of hiring a new employee. It involves lots of time, research, analysis, resource, money and risk. It doesn’t seem to exist an easy recipe for this task, and for the company’s success it’s a matter of knowledge and experience but also of luck — it’s almost impossible to predict a human being, so the company can hire a genius who turns out to be a very difficult person to deal with. That’s why lots of companies outsource this task to another company that is specialized in finding the ideal employee. But capitalism is competition, which is synonymous with lowing costs in order to grow and be relevant on the market.
As software programming seems just a matter of logic and computers deal very well with logic, why not select candidates automatically, bypassing most of the time-consuming tasks and then doing a simple live interview only with those who have succeeded the automated assessments? By doing so we can save lots of money, right? Well, this is the scenario where automated recruitment has emerged. It’s growing so fast that today it seems reasonable just because of continuous repetition. But like any other repetition, it doesn’t turn to be true just because it’s being repeated by many.
In a nutshell, automating the recruiting process is a myth because machines can’t analyze the social factor of human beings. Well, at least not at this moment. An employee of the 21st century isn't the same as Chaplin’s Modern Times (1936) character who works in an alienating production line, repeating such small tasks that the brain is almost turned off. Most of today’s jobs require not only technical skills, but motivation, creativity and, especially, a good relationship between employees. Failing the human factor is failing the whole process, even if the technical factor is impeccable.
The series Mars (2016), from National Geographic, made a bunch of interviews with specialists in the fields of astrophysics, astronomy, engineering, psychology, etc., all working together in a global consortium aiming the sending of humans to the Red Planet, and there’s a consensus among them: although this mission requires billions of dollars and technologies that we have yet to create and improve, if during the trip an astronaut freaks out or simply doesn't get well with another astronaut, the mission could simply fail. They’re not talking about pushing the wrong button or accidentally opening the hatch, but social problems that can potentially compromise the mission. They know that the human part of this mission is as important as the technology one. That’s why they’re investing in experiments involving small groups of people isolated for months. The results so far are not exactly exciting. In a recent Russian experiment called Mars-500, a group of 6 astronauts was kept isolated from almost everything for around 500 days. No internet, no phone calls and just a twice-a-day file upload that could be forwarded by mission support to the participants’ families. After some time only 2 of them kept their minds healthy, while the others suffered severe insomnia, impulsiveness, emotion and humor issues.
Of course, programmers aren’t subjected to the same amount of pressure and risk that astronauts are. But the social factor still play a great role here, probably in the same proportion. For example, the research called Relationships @Work, made by LinkedIn in 2014, reveals that keeping good relationships inside work can make employees happier, therefore improving their motivations and working performance. So even if an automated recruiting process could evaluate the technical part perfectly, we would still need the other halfway in order to get an employee aligned with the company’s expectations. Unfortunately, automated recruitment also affects the technical part evaluation.
There are a few kinds of automated assessments. One of the most famous is Codility, among similar ones like TestDome and HackerRank. These websites provide services for both users and companies. Users can test their own abilities on a big number of free algorithm challenges that can be solved in a bunch of different programming languages. And companies can pick up some challenges aiming at a specific level of expertise and give them to a candidate. Each challenge offers a problem and gives the candidates around 30 minutes to solve it. Before submitting the final solution, candidates may have the chance to run some unit tests provided by the website. No one can deny that this approach has some potential.
It can stimulate the candidate to think about an abstract problem instead of just guessing commands in a boring multiple-choice exam. Users can code even without an external IDE/compiler, since the service provides a programming environment that runs inside the browser, requiring no third-party plugin to work — although it has some limitations when compared to a modern IDE, like missing a good auto-complete and refactoring features. Also, as candidates have to write their own code, they can show not only their knowledge on the chosen programming language, but on algorithms in general. From the perspective of the employer there are some benefits too. They have access to a series of automatic measurements, including a larger number of unit tests ran over the candidate’s code.
The main issue with this approach is that it tries to automate something that requires more than a bot to handle. For example, they don’t consider communication skills, like team discussion or even pair programming — which is a very common way of programming nowadays, as NASA, Google, Facebook and Twitter have shown, just to name a few. No evaluating of architecture, design pattern or clean code. In fact, these challenges can be solved without any OO (object-oriented) concept even if you choose an OO language to write your code. By not measuring programming/debugging skills, but academic knowledge like sorting, linked lists, prime numbers and playful problems, it means that lots of challenges can be solved if you have a fresh academic background and little programming skills. It seems to conflict with companies’ belief that inexperienced people can’t code a real software as good as someone who has been working on the market for many years — and that’s why they usually ask for at least 5 years of market experience. So this raises a question: do companies really know what they’re evaluating here?
These tests are usually concerned about performance (typically asking for O(N) complexity!), although most of the time, when programming a real product, performance isn't the main concern — other things like clean code, smart code re-using, readability, architecture, security, stability, usability, planning, etc. are more relevant and unfortunately are not evaluated. But it doesn't end here.
Another concern is the really short time window, which is unreal, especially considering the heavy pressure that the candidates are submitted to during the tests. Such a small deadline blocks the candidate’s ability to think straight, making it hard to even create unit tests, like the ones that the service team made during hours of work. This kind of pressure isn't just a misrepresentation of the real world, it’s also unfair and silly, since it’s known that fast programming produces bad code and works like a time bomb, showing a lack of medium-long term vision.
When facing a good challenge, programmers usually keep it on their minds after work, thinking about it during their way back home, at bed before sleeping or even waking up with an epiphany. That’s because interesting challenges tick something in their brains, pushing them beyond the mundane daily work. And for the enthusiasts, even after finding an initial solution it doesn't seem to be enough; they keep optimizing their codes and discussing with other people until they have a satisfying solution. Good code requires time and polishing.
At least these services can give candidates the possibility of programming and improving their codes. Companies can even play a video of the code as it was typed by the candidate. And the most important thing: companies can see how the candidate thinks even if he/she has failed in achieving the best solution. Unfortunately, there are some worse kinds of assessments provided by other companies, like CEB SHL Talent Measurement.
This kind of service simplifies things to a point that candidates don’t even need to write a single character of code or anything. Just multiple-choice questions! But the problems don’t end here. They seem to believe that the main concern when evaluating candidates is their memory regarding language commands. This Jurassic methodology has nothing to do with our current technology. They don’t see that memorizing commands is something that doesn't require intelligence since anyone can get it quickly just by searching. They miss the main point: elaborating an algorithm is tricky because it requires developers to think, discuss, be creative and flexible and to come up with an abstract and specific solution that probably didn't exist before. Also, if you’re a good developer in one programming language, chances are that you can be good at any other one pretty quick.
They probably have guessed part of the issue in their approach, so they came up with this idea: giving a time limit of just 3 minutes per question, with no possibility to review any of them after clicking next, even if you don’t use all the given time. This ridiculous amount of time creates stress and freezes most of the candidate’s reasoning. Indeed this is not a specific issue with programmers, but with human beings that are known to suck when have to deal with short deadlines. The psychological effect plays a big role here, as noted by this Triplebyte study. If you have endless time to solve a problem, you can finish it in, let’s say, X minutes. But trying to solve the same task knowing in advance that you have the same X minutes automatically slows you down to the point that you probably wouldn't finish the task on time.
If you somehow manage to abstract the psychological pressure, you’ll figure out that you don’t have time to even test your code using your IDE/compiler because the code provided by the service is trapped inside jpeg images, so the moment you finish typing you will probably have run out of time.
Also, since all the questions offer multiple choice answers, if you don’t know you can simply guess. There’s no room for creativity or questioning — imagine a question that is simply wrong and you can’t even argue. And by forbidding candidates to review their answers, the service is invalidating refactoring, known as one of the most important things for good code. What’s the relation of this with good coding?
Last, but not least, a lot of questions have absolutely no relevance in the real world. In many cases, the compiler would show errors so the programmer would guess the problem easily. Also, the available answers are so similar that it looks like they were specifically designed to confuse candidates, instead of checking how good are their coding skills. By the way, there’s absolutely no question regarding clean code, SOLID, KISS or DRY principles.
Lots of good developers who have been working on the market for years can’t pass most of these exams. In fact, many of them all around the world are expressing their opinions about this madness. A big movement started this year when David Heinemeier Hansson, the guy who created the popular Ruby on Rails framework, decided to tweet that he would fail writing bubble sort algorithm on a whiteboard — another common and odd way of evaluating a candidate. After him, other developers picked up the meme idea and started to confess their coding sins. The issue has received a lot of attention, but it seems that the industry isn't listening yet — you can check the Hiring Without Whiteboards list with all companies around the world known for not using flawed hiring process.
But let’s forget about all the problems related to these automated services for a while. Let’s pretend they work flawlessly. So imagine 10 candidates and their grades in a given assessment. Companies say they usually pick up just a few of the best to look a little bit closer. So let’s say the top 3 candidates are selected to go to the next step, like an interview. The company probably has (or at least it should have) an idea of what kind of human being they’re looking for. Now imagine that the desired personality is found on 3 candidates, however it won’t always coincide with the top grades, but rather randomly. Think about it: what’s the chance of choosing a candidate with both one of the top grades and the desirable personality? The conclusion here is inexorable: companies are giving way more attention to a flawed technical evaluation than the social behavior of the candidates.
Although companies have no scientific study to support this approach, they still do that because it seems cheaper and because there’s still a strong echo from the 19th century, where employees were seen as machines with the only purpose of producing more profit to the company. But employees should not be seen as a bunch of numbers in an Excel table. Today we know that the human side of an employee, such as social integration, sense of liberty, space for creativity, feeling of contribution, etc. is the base of motivation. What else a company should expect from its developers, you may ask. What about employees who are mentally stable, empathic, positive, flexible and easy-going? People with some sense of humor, genuine desire for self-improvement, critical thinking and many other characteristics that an automated evaluation simply can’t handle.
To evaluate a developer or basically any potential employee, companies should focus on the human side, the capacity to deal with other employees, preferentially considering the specific team where this person is going to join. However, usually companies describe a role by giving all the details regarding the so-called desired skills, but absolutely no emphasis on the personality or social skills. Nevertheless, inside the business market, programming is not that geniustopia of a single man changing the world by himself, programming through the night with a magic coffee. It is in fact a social activity. So in order to create a good product and keep it constantly improving, it’s better to have a group of average skilled developers who work together organically than two geniuses with absolutely no social skills that can ruin the project out of sudden. Moreover, technical learning is cheaper and faster when compared to social learning. And when a company doesn't believe in the growth of its employees you can notice a light turning on and telling that this company has eyes from the 19th century.
Sure, in order to have a decent glimpse of the human factor of a candidate the company has to come up with creative approaches, and to do that it probably needs a psychologist or someone with good sensitivity, lots of experience and time. There’s no single solution for that. The interviewer can start an informal one on one conversation — more than one interviewer at the same time can be oppressive. Then some mundane questions can be made, such as: what you like to do, what makes you tick, what was the last challenging situation you had to handle, what you would like to know about the company, etc. The interviewer might guide the conversation to avoid it to be shallow so that it’s possible to get a better idea of who the candidate is. The key thing here is that the interviewer should be empathetic and authentic, so the candidate can feel comfortable.
In the technical evaluation, the company probably wants to detect if the candidate has a developer mindset, which is not attached to any specific programming language, framework or tool. According to Lisa Stern Hayne, global staffing lead and senior recruiter of Google, it’s better to have applicants who are smart generalists and problem-solvers over role-related knowledge, because positions are constantly shifting. And since the candidate is probably not applying for just one company, it’s very likely that he/she has already been rejected a couple of times, so it’s reasonable to not ask for anything that is very time-consuming or complex, also because the company doesn’t need that to evaluate the candidate.
A simple project, like a shopping cart, is enough to see if this person writes good code, uses the conventions the company is asking for and provides an overall correct solution. There’s no need to set a short deadline or put an employee to watch the candidate during the test. When we know we’re being watched we change our behavior, especially in stressful situations like this.
Another solution would be having a pair programming session with an interviewer to implement a small feature. So the company can check the candidate’s social and technical skills. Of course, this can be very stressful for the candidate, so the idea behind it is not asking for complex and perfect solutions, but rather just checking how the candidate deals with teamwork and how he/she approaches the solution.
Finally, inviting the candidate to a conversation with the team is desirable, after all the team is going to direct work with this person. However, instead of confronting the candidate, it would be better to conduct the conversation like a relaxing meeting, without any pressure or a specific set of rules.
It’s worth it remembering though that regardless of the methodology, this is always going to be an imbalanced situation, where the company is a powerful entity composed of many people, having not just the right answers in advance, but also lots of resources to support it. While the candidate is just a person, probably unemployed or unhappy with the current job, trying to guess what this new company is really looking for.
Interviews and tests have been so specific and hard to pass that there’s a big market of coaches supposedly teaching people how to nail the process. According to the Triplebyte study mentioned before, harder interview questions actually do a worse job of predicting outcomes than easier ones:
Harder questions ultimately filter out too many qualified candidates […]. Easier interview questions are less stressful, which is an important advantage. Stress causes candidates to under-perform. When candidates are more comfortable, they perform their true best […]. Being able to offer help makes interviews less stressful which, again, results in more accurate outcomes. Our finding is simply that easier questions provide more signal.
If it’s not enough for developers to be constantly updating their technical skills in a fast pace market, they also have to be specialists in this uncertain area of interviews, which has basically nothing to do with their daily work routines. In the article The Horrifically Dystopian World of Software Engineering Interviews, Jared Nelsen tells this real story to illustrate how absurd this can be:
At one particular ‘top’ tech company the process is that when a candidate goes through an interview he or she has a packet compiled about their interview performance. The packet then goes to a committee whose job it is to impartially review the packet to make a hiring decision. At one point a particular committee got so critical that they rejected every packet for several months. When HR caught wind of this they decided to set up a test. They sent the committee a new round of packets and once again the committee rejected them all. HR then called them all into a meeting and explained that the packets they had just reviewed were in fact the hiring committee member’s own packets from when they interviewed for that company. They had unknowingly rejected themselves!
Hiring a new employee can be a really hard and consuming task. But for complex problems there are no easy solutions. At least, by doing something more humane companies probably won’t blow projects or lose good developers. In this regard, it’s relevant to remember the episode involving Facebook and a rejected candidate, that then wrote WhatsApp and ironically sold it years later to the same Facebook for US$ 19 billion, which is the most expensive acquisition Facebook has made so far.
P.S.: in order to finish this article I had to do some research and polish it countless times. Writing a text (especially when you’re not a native speaker) is like coding: it requires time, dedication and a big effort for continuous improvement.
References (I highly recommend reading users’ comments):
MeatNPotatoes — Codility’s Code Monkey Test & Questions on Software Companies
WAQAS, Hamza — Codility for Experienced Software Engineers!
HSU, Elvis — My Codility Test Experience
