How Lunit hires top AI talent

Hiring is really important in any company. The people you hire will determine what the company will look like a few years down the road and new people can either have a big positive or negative impact on the team. Because of its importance, a typical engineer at a tech company spends a few hours to, sometimes, a day a week evaluating candidates during busy hiring seasons.

At Lunit, we have an excellent HR and talent acquisition team that coordinate the hiring process and help us maintain a world class team. However, the actual hiring process for engineers and researchers is typically done by engineers and researchers. In this post, I will discuss some lessons from interviewing hundreds of candidates in two med tech companies in the last five years. I will provide a peek into the kitchen of how we hire at Lunit and how we manage to be one of the top medical tech companies and attract excellent people.

Why hire

Before going into details about hiring, let’s first take a bird’s eye look at why we hire exactly. Companies usually start small and grow by either sales or investment. Growth can mean more work needs to be done. The work that can be performed can roughly be defined as follows (yes, this is a post written by an engineer/researcher :) ):

where n is the number of people in a team and productivity_n is the productivity of person n. Note how each person can vary in productivity.

One way to do more work is to increase the productivity per person. This can be done by having more efficient working processes, like automating parts of the work, improving documentation, adding unit and integration testing to code bases, careful road-mapping, etc. It can also also be achieved through training of people or increasing physical and mental well being of employees.

Another way to do more work is to hire more people. However, having more people does not necessarily mean you can do more work. Just like in parallel processing in a computer, there is some overhead in communication. Working with one other engineer on a piece of code is typically simple, if you work with 10 others you’ll have some communication issues. It also strongly depends on the person and their role in the team. In extreme cases, productivity can be a negative number if a person is not a right fit.

Another reason to hire can be if people in the current team leave. This is a natural process, people sometimes just want a new challenge or explore a different career path where they feel they develop themselves more or in different ways. The rate in which people leave is typically referred to as churn and how much/long people stay is the retention.

In general, retention is more important than hiring. Finding good engineers is challenging already and if someone has been at the company for several years, a new person can not easily replace them. In some cases, even given the same background, it may take a year or more before the person is at the same level of knowledge and productivity. Unfortunately, some churn is typically unavoidable.

Hiring is an investment. Team members have to spend time interviewing and evaluating candidates, which means there is less time to do actual work. Carefully thinking about the hiring process is therefore important. If not done right, it can cost more work than it reduces.

The hiring pipeline

Hiring typically does not start with interviewing, but with marketing. If we want to attract good people, we usually need a large candidate pool (the people that express interest in a job, for example by sending a CV). For researchers and engineers, increasing the size of the pool is done by maintaining presence at conferences, speaking at local events, publishing papers and releasing data and source code. A larger hiring pool typically means we can find better candidates, but also means more time has to be spent on the process.

Interview process

If we have some candidates in the pool, we can start the interview process. This can be seen as a screening procedure, much like screening is done in health care. Like in health care, this is a bit like looking for a needle in a haystack: we have far more candidates than openings. For most roles in our team the acceptance rate is around 1–2%. However, there are also some key differences:

1. What we want to detect is a match: someone capable of doing the job, who is also a good fit in terms of motivation and culture. We typically do not want to detect ‘all matches’ but want the best one or two matches for the job.

2. The interview process is bi-directional: the candidate is also evaluating us. People typically interview with multiple companies and have to decide what offer to eventually accept. They also want to get as much information as possible out of the hiring process.

Much like a screening process, the interview pipeline has different parameters that can be tuned to make this process as efficient as possible for both us and the candidates. These include:

• The number of interviews that are performed for each candidate. This can range from one or two to up to a dozen. Companies like Google and Netflix are (in)famous for their large number of interviews. This comes at a cost for them and also for the candidate and it is not something all companies are comfortable doing.
• The acceptance rate of each interview. By setting a high standard especially at an early stage, you reduce the amount of work for your fellow engineers/researchers and the candidate, but you also risk missing out on qualified people. Some companies choose to perform interviews sequentially, where after each interview the candidate can be rejected. Some companies choose to perform several interviews and aggregate scores from interviewers before proceeding to a rejection or a next round of interviews. A purely sequential process is cheaper, but also more noisy.
• The number of people in the interview. Having multiple people in one interview can be intimidating for candidates, however it can also make the process more objective and less noisy as the opinion of multiple people is combined. Two interviews of 1 hour with two different interviewers costs the same amount of time for a company as one 1 hour interview with two interviewers. It leaves more time for questions, but also takes more time from the candidate.
• The format of each interview. This can be a face to face chat about their work, a presentation about their work, a coding assignment, a math test, etc. This format varies widely per company and there is still lively discussion about what the best method is. Testing this scientifically is also really difficult as it requires large sample sizes and clear job performance metrics for engineers and researchers, which are also contentious.

Out of respect for the candidate and team members, we at Lunit are trying to minimize the amount of time we and the candidate have to spend on the process, while trying to give and get the most information to/about the candidate as possible. The time spent can be modulated by tuning all the parameters defined above. To better illustrate this, let’s have a look at two examples below, which show snippets of a typical engineering interview pipeline.

The pipeline starts with a landing interview or intro call, followed by a take-home assignment and a tech interview. In the first case, we have a high bar for the intro call, and have less people joining the tech interviews. In the second case, everyone gets to do the assignment, but we raise the bar a bit in the tech interview. In both cases, we are left with one candidate.

Assuming the salary of all interviewers are the same, we only have to look at the total interview time to make an estimate of the cost. In the first case this is 100 * 1 + 50 * 9 + 5 * 4.5 = 575.5 hours. In the second case, this is 100 * 1 + 100 * 9 + 10 * 4.5 = 1081 hours. The second process is more costly for both the company and the candidates on average, but also more rigorous and can result in a better match in the end, because we spend more time with the candidate.

Optimizing signal

As mentioned before, during the interview process, we want to get the most information about the candidate. We also want to provide the candidate with as much information about the job as possible. We refer to this as ‘signal’. The information should tell us how good a match the candidate is for the job and our job is for the candidate. So what is a good match?

What is a good match?

There are many aspects that have to be evaluated to determine if someone is a good match. This would include soft skills like communication, attitude for growth and grit. However, these hold for all jobs and since this post is about engineering and research roles specifically, let’s have a look at a few specific items.

For most engineering and research jobs, raw problem solving skills and the ability to learn new skills are probably the most important. Especially in the world of AI, things change rapidly and making quick adjustments/obtaining new information is key. This is especially important in early-stage startups and small companies where roles and responsibilities are broad and fluid, and a person usually has to wear many hats.

It is uncommon for someone to have all the skills needed for the job. However, it is important that the candidate has some skills that can readily be applied to the job. For example, someone may not have experience with medical image analysis but have a large amount of experience in developing real-world machine learning applications, the rest can be learned on the job.

A common bias is evaluating candidates based on your own skill set. For example, I once heard of medical imaging candidates being asked to implement a linked list on paper for a research position in a medical imaging company, because they were interviewed by an inexperienced team of developers from an unrelated field. In a team, it can be good to have some diversity of skills and thoughts. If everyone has the same skill set or way of thinking, you will get a weak ensemble. Look for skills that you know are needed for the team but the current team is lacking.

What to ask?

As mentioned above, problem solving skills are key. Raw knowledge can be Googled nowadays, but some basic knowledge should still be expected, else the ramp-up time would be too long and it is not practical to Google everything all the time. During the interview process, we want to test some basic knowledge but then focus mostly on problem solving.

Getting an understanding of someone’s knowledge and skills in a short time is challenging. The interview can be seen as a sampling process. If you have limited time, you may just get a small and bad sample. Asking a candidate a few facts is typically not so useful. Open-ended questions that provide the candidate some freedom will get you more signal.

For example, instead of asking a question that has a single answer, like

‘How does Python compile the source code’, ‘what is batch normalization’, etc

you could ask

‘What do you think is well-designed about Python’, ‘What are some differences between Python and C++ that you think make Python more suitable for data science’, ‘Could you explain a few basic network modules that you like to use’, …’ etc.

Open-ended questions/problems

To test someone’s problem solving skills, my personal preference is to focus on real-world problems that I faced myself. This will simulate how it would be to work with this person on this actual problem. For the candidate, this is also good because they experience what it would be like to work with you.

For example, a while back I trained, what I thought was, the same model on two different machines. However, the resulting model and detection performance is different. What is going on? Going through the debugging process together with candidates helped me get an understanding of what it would be to work together.

Roles and seniority

The type of questions one would ask are different for different roles. Some teams distinguish between I-shaped and T-shaped engineers, where the I-shaped person is an expert in a single area and the T-shaped person has moderate-depth skills in multiple areas and in-depth skills in one or two. Sometimes other letters like Y and X are added to define other profiles. Early-stage startups may prefer broad skill sets and flexibility, and people in senior or management roles with interdisciplinary teams may also need that.

Not only the role, but also the level of seniority should determine how to set up the interview. Senior roles are typically more difficult to hire for, firstly because the candidate pool is usually smaller. Secondly, the work senior people do is more high-level (road mapping, architecture of code, mentoring, etc) which is more difficult to evaluate. We can ask someone to implement and analyze the complexity of a search algorithm and the correctness can quite easily be validated. However, a high-level software architecture or roadmap is more difficult to judge.

Assignments and on-site days

Another good way to test problem solving and to simulate what it would be to work with this person is by providing a take-home assignment or inviting them to work on-site for one or two days. The latter would require the candidate to take some time off, which is not always possible. If the assignment is a good real-world example of what we work on, this again is also a good opportunity for the candidate to get familiar with what the job would be like.

Assignments should be challenging and limited in time to get the most information out of it and out of respect for the candidates’ time. They should have clear deliverables, but enough freedom for the candidate to give their own interpretation. If the assignment is too simple or too well defined, every submission might be the same and it will lack discriminating power.

Questions from the candidate

Last but not least, it would be good to leave ample time for questions from the candidate at every stage of the interview process. As mentioned before, the interview process should be bi-directional. The candidate should also have a good idea of what to expect. They typically interview with multiple companies and need to make a good decision about their future career. The questions the candidate asks can also teach you a lot about their motivation, prior knowledge of the company and mode of thinking.

It is best practice to answer any questions as clearly, honestly and as realistically as possible. Especially if the person is a must-have, it can be tempting to oversell the company. However, this is typically not good long term. Satisfaction = reality — expectation. If the candidate joins and ends up being disappointed, it will be bad for both the company and the candidate.

Conclusion

The hiring process is complex and noisy. However, we do our best to make the experience as pleasant and efficient as possible. Unfortunately, given the low acceptance rate, we often have to reject many qualified people. Sometimes the reasons for rejections are not related to a person’s skills or the interview process (e.g., a person may seem overqualified, there is change in budgeting, the job is filled out internally, difficulties in arranging visas/salary expectations, etc.). The most important part about the whole process for us and the candidate is to keep a positive attitude and keep interviewing. Eventually the right candidate and right job will be found.

Acknowledgement

Thanks to Jussarang, Sergio, Sora for feedback and suggestions on this post (in alphabetical order).

References

Some of the ideas in this post are inspired by the following sources

[1] An Elegant Puzzle: Systems of Eng Management — Will Larson
[2] Work Rules!: Insights from Inside Google That Will Transform How You Live and Lead — Laszlo Bock