Intern Interviewing Re-Invented
At Box, we hire returning interns for most of our entry-level engineering positions. This makes the hiring process for interns more important than you would think. It’s not “just an internship”, we’re hiring our potential new team members!
In re-inventing our process on intern interviewing, we had three goals: 1) decrease Boxers’ time spent on interviewing 2) increase acceptance rate and 3) increase diversity. We more than doubled the offer acceptance rate, reduced engineering time spent on interviewing by 40%, and hired half of our interns from underrepresented groups. We saw a similar increase in the actual intern to full-time offer and full-time offer acceptance rate. Through some strategic process changes we were able to achieve our objectives. Read on to learn how!
Before we changed the process
Previously, we had an interview process with a timed online coding exercise and multiple technical phone interviews. Our engineers spent a lot of time doing these, with only few candidates passing the bar and getting an offer. The Silicon Valley is a very competitive place to hire competent, skillful and culture-add employees. Some companies could be more attractive than others for interns, just owing to the brand awareness they need to build as they sell to customers directly. As an enterprise software company, Box faces the challenge of not always being a tool college students use or are familiar with. Usually we gained a lot of traction once potential interns learned more about Box’s culture and software. But, this was often too late in the hiring process and we were losing to companies that folks were more familiar with from the start. This lead to a higher than ideal decline rate, and more hours spent interviewing.
Trying something new…
Three years ago, we formed a tiger team of university recruiters and engineers to re-invent our intern interviewing process, and we have been iterating on process improvements ever since. I participated as an interviewer in year one, and became so passionate about the topic that I joined the tiger team in year two. We’ve made some meaningful changes over the years, and I’d like to share our learnings, as well as the results we’re seeing.
Take home assignment strategy
We replaced the technical phone interview with a two week take-home assignment. During those two weeks, we offered office hours with our engineers to provide guidance on the assignment and had open Q&As about Box and being an engineer at Box. After the two weeks, the candidate would get a phone call with a recruiter and a video call with one of our engineers to do some pair programming extending their submission.
But wait, you might wonder, isn’t that much more work? For both candidates and the interviewers? What busy student has time to work on an extra assignment for two weeks? And the engineers still need to conduct an interview with every candidate on top of hosting the office hours!
We had a hypothesis: if we moved from a rigid, test-based environment to a more flexible, project-based environment, this would yield stronger results from a more diverse set of individuals. Project-based evaluation includes two key factors that are self driven: 1) time prioritization — e.g. how much energy gets spent on different components of this project, and 2) using information resources — e.g. if I don’t know how to solve a specific problem, students can find resources, research, ask questions etc. This is more similar to how problems get solved in our working environment, and lower stress for many students who face test anxiety.
The two week take-home assignment
We gave out the assignment in cohorts, so a group of students would start at the same time. If someone wanted to participate, but knew that a particular week would be very busy because of exams, other school work, or a trip to visit family, they could ask to join in a later cohort. The assignment was also designed to not take two weeks of full-time coding. In the first iteration our estimate was about ten hours. In later iterations we also provided some boiler-plate code and utility functions to shorten the required time.
But even with this flexibility, we were asking for more of a time commitment from students than with the typical phone interview. To balance that out, we designed the process in such a way that participation — no matter if it resulted in an offer or not — was valuable for the applicant. The assignment put an emphasis on understanding the requirements, design, object-oriented programming and self-directed coding, which is in many cases more challenging than typical school assignments and therefore a strong learning opportunity. We got feedback from students that they enjoyed working on the project and felt it gave them the opportunity to show their skills in a way that traditional interviews don’t allow.
The format that’s based on skills actually used in software development also means the students don’t need to spend time specifically practicing for whiteboard or brain-teaser coding interviews.
“It was a fun project to do and of a reasonable scope so it felt well planned out and interesting, as well as more representative of how I work than a more traditional interview would have been.”
“Although it took quite a bit of time, it was really fun making something from scratch as well. I believe it allowed me to show my skills and dedication much more than any 2-hour coding challenge possibly could.”
“Although the take home exercise was difficult and took time, I felt that it was the best way to have a fair evaluation of my skills and experience. Furthermore, I was able to practice coding skills that I’m not usually able to and debug unique problems.”
Office Hours
The office hours were another building block in making the interview process more valuable for the students. Each office hour block consisted of two parts. First a short presentation, then an open Q&A with Box engineers. During the presentations, we provided guidance on the process — how to get set up, how to use the test runner, clarification on the requirements, and how to make the submission. Each presentation also included general advice for working on a software project. The topics ranged from how to use git, to design principles and object-oriented programming.
We also presented some information about Box history, culture and engineering and included open Q&A where the engineers shared stories about their work, their experiences at Box, and their own internships. This let the students get an idea of what to expect from working as a software engineer and from working at Box. We hoped that the information could help any student to get a better idea of what they might be interested in doing with their degrees, while also getting them more curious and excited about being a software engineer at Box.
“I liked how we get to learn a lot about the company while doing the interview process and what we should expect working at Box, and what type of projects we would be doing.”
“I really enjoyed the office hours. It really made me excited about the opportunity since I learned a lot about Box’s culture through the Boxers that hosted office hours. I feel that those office hours were one of the biggest contributors to my excitement about the opportunity.”
Coding interview & behavioral interview
After the students submitted their projects, it was time for the recruiter call and the video call with an engineer. In the recruiter call, they talked about the students’ experiences, challenges and goals. During the technical session, an engineer asked the candidate to expand their code. This allowed us to see how well the candidate could communicate about their work, how flexible the design was, and weed out potential plagiarism. As with the other parts of the process, it was our goal that the candidates learn from this experience, regardless of getting an eventual offer from Box, so the engineer offered advice, feedback on the design and debugging assistance during the session.
One key difference made this type of technical interview much more enjoyable for most people than a typical technical interview: The candidate got to work with and talk about code they had already written, and worked in a problem space they were already familiar with. This could make a huge difference in regards to nervousness and comfort-level during the interview. And, it’s a situation that’s much closer to real work experiences. How often do you get put on the spot to immediately come up with a solution to a completely unexpected problem? Not often. Instead, it’s much more common to do a code-walkthrough, or a pair programming and debugging session with a co-worker on a topic you’ve already spent some time thinking about and working on.
“The code walkthrough was extremely helpful in my learning experience and the engineer was extremely personable and interesting to talk to.”
“I also really liked the code walk through for the take home assignment. I was able to discuss the design decisions I made and show how flexible my code was!”
“I also thought the interview after were we had to make changes to our code’s functionality was super fun and definitely the most enjoyable interview I have ever had.”
What we got from doing it differently
Having a more involved interview process also had the effect that candidates self-selected based on how excited and interested they were both about the project and the position at Box. When the time came for the technical session with an engineer, most students in the process were very seriously considering the internship at Box, especially after learning more about the company, getting to know the engineers, and putting effort into their project. The information from the office hours gave students a much better idea of what to expect during the internship and working at Box, and therefore the expectations on both ends were better aligned.
This manifested in an increased offer acceptance rate. There were fewer students who interviewed just because “why not?” without being really interested. So, the overall Box engineering time spent on interviewing went from 292 hours with the old process, to 175 hours with the new process (and that’s including the time spent on office hours and reviewing the code before the technical interviews).
When looking beyond the intern interview process, it also appeared the new process helped find better qualified hires with better aligned expectations. There were two numbers we looked at: 1) intern offer acceptance rates, and also 2) intern conversion rates (how many new hires received and accepted a full time offer from Box).We extended more full-time offers to the interns than in previous years, and we saw a similar increase in the FTE acceptance rate as with the internship offers. There was overwhelmingly positive feedback from both the internal teams and hiring managers, and also the interns we hired. They were exceptionally motivated and — based on the numbers — had a better time than interns hired with the old process. We believe that this was due the new process giving students a better idea of what to expect, so that any mismatch in expectations got resolved and eliminated early.
The process also became helpful training for our current Box engineers helping with interviews in two ways: 1) business leadership skills — helping solve ambiguous, not purely technical business problems, and 2) mentorship skills — providing the opportunity to build relationships with future teammates. A lot of our interviewers for the program were engineers early in their career, and for some of them it was the first opportunity to get involved in the hiring process. We saw a lot more enthusiasm and engagement from the interviewer side than in regular interviewing. The office hours and coding sessions were a lot of fun, and many engineers came back to the process in the following years to participate or then join the tiger team (like I did) and lead the process. We specifically searched for interviewers from teams that were interested in hosting an intern in the coming summer, so often interviewers from those teams would also go on to become mentors for an intern who decided to join their team.
Removing barriers
Given that most of our entry-level engineering positions are filled by returning interns it’s important to keep diversity in mind throughout the process. There were a lot of things done by the recruiting team to make sure we reached candidates from diverse backgrounds, like reaching out to various colleges and partnering with student organizations to get our job posting out to all students. But we also strived to make the hiring process throughout as accessible and unbiased as possible.
For the preliminary online coding test, we changed the “packaging” or framing of our questions to a common Box theme, as opposed to random topics that could make the question confusing for someone not familiar with the topic. One example is a question from previous years that had a “Fantasy Sports League” theme (I didn’t even know what that was). Even if the underlying coding problem of the question is common and standard, packaging can negatively impact performance when trying to solve the question under time pressure.
We examined what exactly we were testing for with the questions and tried to avoid anything that might be selecting for skills or knowledge that aren’t relevant. The purpose of this stage of the process was to ensure the candidates have a minimum ability in coding before moving on to the take-home assignment. We wanted questions that could be solved by identifying the right data structures to use and applying the correct algorithms, and avoid questions that require an aha-moment to solve or that were super-easy if you’ve encountered that particular problem before (but nearly impossible if you haven’t).
We tracked success rates and compared them with previous years, to see if our changes moved things in the desired direction. Previously, the pass rate of female candidates was only about half of the pass rate of male candidates, but with our adjustments, the pass rates moved into the same range.
The two week take-home assignment combined with the coding session was based on the same idea of better testing for what we actually look for in a candidate. It allowed us to cover a lot more aspects than a single technical interview on the phone. With the submitted project we could evaluate how well the candidate understood the requirements, design decisions, correctness, writing clean and easy-to-follow code, while the coding session gave an impression of how well the candidate can communicate, work with their own code and how flexible the design was. By looking at more data points provided by the take-home assignment, we could evaluate each candidate more reliably and on multiple dimensions that we believed were most relevant to a successful internship.
We also wanted to keep in mind the candidate experience: by letting the applicant work with their own code and in a problem space they were already familiar with, we reduced the nervousness and surprise-factor that often comes with typical coding interviews. An interview that just throws an unknown problem at the candidate favors people who deal well with being put on the spot and instantly adjusting to a new challenge. We aimed to make the experience comfortable for all types of personalities, including those less successful with traditional test environments.
“I also enjoyed the take home assignment because it took the pressure away from having to complete a timed challenge and allowed me to showcase my abilities.”
“I really liked the idea of the take home project, and I do really feel like it showcased my skills as a developer more than a live technical interview does. Especially because I know how horribly I can do during those, forgetting the most basic computer science concepts. So I felt like this eased that pressure.”
“It was really nice to have the opportunity to think about my design and code and I felt like it evaluated my coding abilities in a way that no other company did. The project made me want to work for Box because I felt that you were one of the only companies who were really willing to give me a shot.”
“I really liked the internship process because it tested people on design and project architecture rather than just a programming question that people could have been unlucky on.”
Conclusion
By changing our intern interviewing process, we made the process much more engaging and rewarding for the applicants. We created more opportunities for the students to learn about engineering at Box and the internship we offer, so the offer acceptance rate increased and our interns had a much better idea what to expect while also drastically reducing our time spent on interviews. At the same time, the new approach removed some of the barriers that come with typical engineering interviews, and we were able to increase the overall diversity of our intern class.
If you’re interested in working with us, check-out our opportunities. We’d love to hear from you! https://careers.box.com/c/engineering-jobs
