Reducing Bias in the Interview Process

But I’m not biased!

Well… you probably are. There’s even a short test you can take to prove it. It’s called the Implicit Association Test and it was developed at Harvard.

The best way to counteract bias is to develop processes to remove subjectivity as much as possible. In the landscape of software engineering, where there are countless areas to improve, where do you even start? This post covers one area — the interview process.

This is what our standard interview process looks like. It is long and takes a bit of time, but at the end, we have exceptional new coworkers.

1. Resume review
2. (Phone) Recruiter screen
3. Skills exercise / homework
4. (Phone) Hiring manager screen
5. (Video) Core values interview
6. (On-site) technical interview
7. (On-site) peer interview
8. (On-site) top grade interview
9. Reference check
10. (Phone) Functional leader call

As the hiring manager for the QA team and Support Engineering team, I come in at step 3 — the review of the exercise. About a year and a half ago, I started getting serious about how to ensure that we get the best candidates while removing the potential for bias along the way. It takes extra work, so I set out to prove its value on the teams I’m directly hiring for, hoping that it would prove easy and successful and the rest of the engineering team would pick it up.

Standardized Scoring

We give the same QA exercise to every QA candidate, and the same coding exercise to every Support Software Engineer candidate. I met with both of the teams, and we came up with evaluation criteria. What were we looking for in this homework? What is required, what is above and beyond? What would be a disqualification? By standardizing our scoring criteria, we’ve set concrete and objective goals that we can easily say yes or no to. We’re not relying on a subjective experience to move the candidate forward.

Anonymization on the QA Team

One of the best ways to reduce the chance for bias is to remove identifying information from the submission. This has been done in blind auditions for orchestras with positive results for increasing the percentage of female musicians. I set out to try to mimic this in our offline exercise review process.

The QA team was easy, although a little painstaking. The exercise is usually submitted either through Google docs or Microsoft Word/Excel. I create a copy, remove or replace any instances of their name, and then rename the file with their initials only. I’ll then send out an email to a team member to review it according to our evaluation criteria.

Things to Watch Out For

Every once in a while someone submits a pdf. If I wasn’t trying to anonymize it, it’s great! What an admirable trait in a candidate— they know it will work no matter the operating system or software I have. In this process though, it requires me to print out their assignment, find a black marker, black out their names, scan it back in, and then email it out. I have to tell myself those extra 10 minutes are worth it; everyone deserves the same chance.

Anonymization on the Support Engineering Team

On to an even bigger challenge! Part of the support engineering exercise is to submit a solution to a coding challenge. Nearly everyone submits a Github repository. This has the added benefit(?) of letting us see the rest of the code they’ve committed into their public repos. That could be good but… that also invites bias in. What if they have only worked for companies and don’t have side projects public? What if they don’t have time at night to code for fun? What if they’re just starting out and you only see Hello World in three different languages? What if you see some really awesome projects, but it’s unclear that they were one of many contributors?

Our support engineering role is our entry-level developer role. We’re looking for people passionate about growing their skills, who have a foundation of full-stack development and a hunger to learn more. Sometimes this comes as a mid-career change straight out of a bootcamp. Sometimes this comes fresh out of college. Sometimes it’s a person looking for their second engineering role after the first hasn’t been great. Given the population of entry level developers is much more diverse than those 10+ years in, I especially wanted to make sure we could reduce bias as we started reviewing these candidates.

There was a recent study where researchers found a gender bias in open source programming. Programmers who could easily be identified as women based on their names or profile pictures had lower pull request acceptance rates (58 percent) than users who could be identified as men (61 percent). Knowing this, I wanted to keep anonymity for our candidates as long as possible.

Anonymization Process

The easiest way to do this is to clone their repo and upload it as my own. However, that removes the commit history, and part of our review process is to see how the candidate thinks, making small commits with helpful comments. If we lose that, we lose the insight into their work process.

Erica Stanley helped me find a tool that someone had already written to account for this. We found https://github.com/cvortmann/git-anonymize and she made it work for us.

Installing the Anonymizer Script

1. Clone the git-anonymizer script: git clone git@github.com:emstanley/git-anonymize.git
2. CD into the newly cloned directory and enter: make install

Anonymizing Candidate Repositories

1. Clone candidate’s repo: git clone git@github.com:[theirusername]/[their_repo].git (can copy this command from their repo)
2. CD into newly cloned directory and enter: git anonymize
3. Run: git log, to ensure commits are now anonymous
4. Remove origin by running: git remote remove origin
5. Go into your own Github, create repo with anonymized name (I use their initials)
6. Add anonymized repo origin (you can copy this command from Github after you create the repo) git remote add origin git@github.com:[yourusername]/[their_repo].git
7. Push the anonymized repo git push -u origin master
8. Refresh page
9. Share the anonymized repo from your own account

Things to Watch Out For

Again, there are candidates out there that are very thorough and put their name in their readmes. Usually this is great, but it can trip up an anonymization process. I search through their readme for their name or other links to their work and redact them. This will mean that their repo will also have a commit from me if I don’t catch it prior to adding it to my own account.

That’s a Lot of Process

Now that I’ve found myself interviewing people and building teams, I realize that by changing processes I have the ability to truly make a difference, and that makes it worth the extra steps. The work put in makes sure that we’re getting the best candidates in for an office visit.

To be clear: there aren’t quotas I’m looking to fill. I wish we had more female engineers and I wish our teams mirrored the diverse population of Atlanta, but I only take part in certain parts of the process and don’t have control over the candidate pool. So we need to find as many people as we can, get as large of a pool as possible, and then pick the best.

Naysayers

We’re going to meet them in person eventually and there could be bias at that point— why bother with this?

Rebuttal: The point is we may not meet them if they can’t pass the offline exercise. There certainly could be bias happening in the onsite interviews, but we can start to learn to counteract it. I would encourage everyone to watch this video from Facebook on Managing Bias.

The rest of their repositories are important. We want to make sure they’re consistently growing.

Rebuttal: There is insight to be gained, but people can grow in a job where they aren’t making changes that can be publicized. It’s also not true that to be a great software engineer at SalesLoft you have to spend all your free time coding on side projects. Helpful? Yes. Required? No.

What if we say no and then they were just having a bad day or they were rushed through it?

Rebuttal: After we make a decision on a candidate, I share their full Github repo with the team. If they didn’t pass the offline exercise, reviewers are free to peruse the other repos and see if there’s anything present that would change their minds. To date, we’ve never changed our minds.

Results

Nothing is perfect, but I think I’ve had a lot of success with these methods. I’m certainly happy that we have this process in place, and as SalesLoft continues to grow, we continue to search for the best and brightest. I’m hoping this small change in the process could be the extra help needed to keep our team operating with such high performers. My priority has never been to build a diverse team — I’ve set out to build the best team.

Pictures of two of the best teams at SalesLoft:

QA Team. Front row: Mo Islam, Belinda Goodman, Johanna Park, Fora Shah, Carina Gerry, Carol Truong. Back row: Matt Coglianese, Fernando Estrada, Jeff Thoensen, Ron Wabukenda. Not pictured: Ted Williams, Kisha Robinson.

Support Engineer Team. front row: Ivy Kroncke, Aylin DeBruyne, Alicia Barrett. Back row: Jimmy Alford, Marco Dell’Olio