Revamping Engineering Interviews- How We Made Our Process Better

Bahar Shah
Sep 4, 2018 · 7 min read

Issues with software engineering interviews are not new or uncommon in the technical world. Here at Bluecore, some of the issues that plagued Bluecore engineers when it came to interviewing happened when we had a normal post-interview sync on a candidate and everyone rated the candidate as a “hire” with no major flags raised, but because no one felt strongly enough about the candidate to rate them as a “strong hire” we ended up passing on actually giving an offer. We found ourselves asking the question: “Why did our rating process no longer match up with the actual qualifications we ended up making our decisions on?” Many readers may find themselves asking similar questions about their interviewing processes as well.

Note: Every company will have their own processes and interview systems in place. What worked for us might not necessarily work for you. The idea here is to explain some of the issues we found ourselves faced with over time and how we came together to evaluate and improve them. This is by no means a definitive process — we will continue iterating and improving upon what we have already done.

Search for technical interviews and you will see that there are lots of issues

At Bluecore, we traditionally have five onsite interviews (both coding and non-coding) followed by a post interview debrief. However, what worked for us when we were a team of 10–20 engineers no longer applied, since we now needed to better accommodate training our interviewers, having specialized interview loops, using different formats for interviews and even having culture focused slots. We could no longer function at a level where every engineer could automatically conduct any interview and have the rating and post-interview outcomes be consistent. We decided to revamp our interview process from the ground up and align on what we wanted as well as improve the process for both the interviewees and the interviewers.

An engineering-interviewer-led taskforce surveyed the engineering team to help focus on ways to improve the interview process in order to focus on impact both technical and cultural) and remain congruent with the overall hiring process at Bluecore.

The interview slots

The evaluation of our interview slots included renaming them to make the purpose clearer to the interviewers, creating rubrics for what we are looking to evaluate during the slot and listing sample questions as examples. Once we agreed on the format of the revised slots, we then presented them to the entire engineering team and led in-depth trainings for each slot. Those new slots include:

Phone Screens

Effective phone screens are the first line of defense in screening out candidates and the first line of offense when it comes to getting a candidate excited about the opportunity. Mismatches in expectations as well as time spent focusing on the wrong areas lead to candidates that advance through the interviewing pipeline without being adequately evaluated or informed in some cases. The most important change we implemented was making it very clear that the focus of the phone screen is on problem solving and how the candidates approach complex questions rather than just evaluating whether they obtained the absolute correct answer on their first try.

Don’t ask questions that have nothing to do with the type of work engineers may encounter at the company

On-Site Interviews

Once candidates are brought on-site, we have them go through a pretty standard set of interview slots (with some variance for specialized loops like Frontend Engineering, SRE, QA, etc). As part of the reevaluation of each interview slot, we categorized what exactly we were looking to gauge from each one. This helped inform the format as well as the types of questions we now ask in each slot. The following are some of the slots we iterated on:

CS Fundamentals -> Whiteboard and Analysis

This slot is our traditional whiteboard-based CS evaluation where we look to see how people approach a problem and whether they can code and analyze their solutions.

In our previous CS Fundamentals interview, it felt like we were just focused on seeing if an interview candidate could recall and use some fundamental concepts rather than considering the candidate more broadly. Now, we want the questions to lead the candidate to throw out solutions, ask questions and think out loud. One way we do this is by asking open-ended questions, which gives us the ability to stretch the question in different directions. Additionally, we now test the candidates’ ability to analyze their own code by evaluating trade-offs, fixing bugs and accommodating new information. We want to make sure we see how they think through options.

From the interviewee experience, they don’t want to be asked about obscure data structures or algorithms they haven’t used since their college days. The biggest problems with most coding interviews today is that they evaluate candidates based on their knowledge of academic computer science rather than the skills and types of issues encountered in the average workday of software engineers.

Real World Problems -> Let’s Build Something

This is another coding slot for us during our interview process. The expectation here is that candidates will be asked to answer a question that more directly aligns with actual issues faced at Bluecore in the past. The issues we ran into were that the types of questions and the format (whiteboarding) we were using didn’t give us sufficient data on how candidates were actually able to work in a realistic environment.

Our solution for this slot was to implement a mechanism by which candidates would actually be coding on a computer in an IDE since it is realistic to:

  • write code on a computer

These are all a daily part of engineering at Bluecore, so we wanted to bring this into our interview process. By having partly written code that candidates now need to go in and modify and extend, we can evaluate a whole new set of skills that are vital to being a good developer. We can see how people approach debugging and testing in an interview setting as well as how proficient they are with reading and grok-ing someone else’s code,which is a good litmus test for anyone who will be joining an existing codebase. By having different engineers write different questions and problems in multiple languages, we now have an interview bank of questions to draw from that allow us to cover most of the candidates that come on-site.

Culture Add -> Core Culture -> Working at Bluecore

The purpose for this slot has always been for us to evaluate if the candidate will be a good addition to the engineering team outside of their technical skills. It’s more than just seeing if a candidate “fits” the culture — we really try evaluating how they will add to the culture that we have cultivated.

Our engineering culture has evolved to encapsulate some of the following paradigms concerning our engineers and their approach to work:

  • Build things as simple as possible and as powerful as necessary

With these values in mind, the main change to this slot was creating a rubric of areas to focus on as well as types of questions to help shine light on certain qualities and provide clearer insight into how to evaluate responses. We also enforced our interviewer shadowing for this slot so that everyone is properly trained in it since we think it is just as important as knowing how to conduct a technical interview.

Lunch

Our interviews take up to five hours. Lunch is a good break in the middle and gives the candidate a chance to meet Bluecorians from all the different departments (Bluecore has free lunch for everyone every day!). We decided to make this slot flexible and have the candidates help drive the direction of the lunch interview by giving them the option to either join the engineering team at lunch, attend a tech talk if it is happening, sit with other parts of the company or go back to a conference room and talk in a 1–1 situation to answer any questions.

Things to keep in mind

Don’t forget the interviewer is also the interviewee- you are also trying to sell the candidate throughout the whole process

Remember that story I told at the beginning of this post about how we had a lack of decisiveness in our feedback and rating process? By having clearly defined expectations and detailed interview training for each interview type, our interviewers are no longer unsure about how to rate candidates. What we evaluate actually lines up with what the interviewers are looking for and everyone is on the same page now.

We set out to make engineering interviews better at Bluecore. What we found along the way is that the interviewer experience is just as important as the interviewee experience. Engineers that have joined after going through the new process have rated it highly and our interviewers have communicated that they feel more prepared to lead interviews. Having clearly defined objectives and questions for each interview slot means that interview results are more standardized, candidates get asked a variety of questions and the position being filled more closely aligns with the process of actually filling it.

Bluecore Engineering

Go behind the scenes of the retail marketing platform. Interested in working with us? Check out our careers page here: https://www.bluecore.com/careers

Bahar Shah

Written by

Software engineer @ Bluecore by day. Baker, reader and wannabe film critic by night.

Bluecore Engineering

Go behind the scenes of the retail marketing platform. Interested in working with us? Check out our careers page here: https://www.bluecore.com/careers

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade