Bringing structure to the engineering interview

Co-authored with Dan Fike

Interviewing engineers is hard.

After only a couple of hours with a candidate, you are expected to confidently reach a conclusion about whether to hire them and how to level them. It’s impossible to get this right, but we have recently restructured hiring at Carta to be less wrong, more fair, and allow us to easily make incremental improvements. We seek to answer the question “What is the highest level we can bring someone in at and see them do great things?” We make leveling decisions before we make yes/no decisions.

Carta has done a lot of hiring lately. In just a few years, our Engineering org has grown from about 40 to 400 people. Scaling like this requires structure and process — leveling engineers ends up being tied to compensation and it becomes increasingly important that we make those decisions in a consistent and fair manner. This was why Carta started leveling candidates via a hiring committee. Our new interview process goes a step further.

We no longer ask our interviewers to make hiring or leveling recommendations; there is too much variance underpinning those conclusions. Our interviewers are now data gatherers, measuring the attributes of each candidate against a standard rubric. Our outcomes are driven more by the candidate than the interviewer. Our process is iterated more by data than intuition.

We can use historical data trends to support our decision-making process

Hire for strengths

Carta is committed to hiring for strengths, not lack of weakness. Our CEO, Henry Ward, discussed this commitment all the way back in his 2016 blog post How to Hire. Much has changed since then, but we kept this idea front and center when we revisited how we interview engineers. We identified the areas of strength that serve as evidence of a candidate’s ability to succeed and thrive as an engineer at Carta. We transformed our findings into the 15 candidate attributes that underpin our interviews.

Enumerating these attributes gives us a few things we didn’t have before:

• A granular understanding of the candidate’s strengths
• Focus on signals that differentiate the most senior candidates
• A consistent vocabulary for describing a candidate’s strengths
• A framework for synthesizing a candidate profile

The attributes act as a checklist for our interviews, providing candidates multiple relevant ways to show their strengths. Our interviewers use the same attributes to structure their feedback. As Carta’s needs evolve over time, we can iterate our attributes to ensure they are formally reflected within our interviewing process.

Structure with rubrics

While our attributes define Carta’s needs, our rubrics measure a candidate’s performance. Attributes and rubrics together from the bedrock structure of our interview and bring us toward patterns and practices proven to work.

Consider a candidate whose skills are aligned with the title of senior engineer. An interviewer with higher expectations will be disappointed. A second interviewer with lower expectations will be impressed.

These interviewers are making decisions based not on the candidate’s performance, but on how that performance compares to their individual expectations. This isn’t how you’d want to be evaluated. It isn’t what we want either, so we don’t ask interviewers if the candidate “passed” or “failed.” Instead, they summarize the candidate’s performance by consulting a set of attribute rubrics like this:

By creating explicit rubrics, we increase consistency in how we construct our interview questions and reduce the likelihood of bias in our interviewers’ feedback. (We aren’t experts in bias, so we involved our DEI team to help.) The meaning of “software design” now has a shared definition for every interviewer.

At the conclusion of each interview panel, the hiring manager and hiring committee have a structured set of data to make their decisions.

Level with data

We don’t try to determine a candidate’s level with some magic formula. Since each attribute has a bespoke rubric, not every incremental step up carries equal weight. Some rubrics are designed to differentiate between higher bands of seniority, while others focus on less-experienced candidates.

Evaluating more senior candidates is even more nuanced. While we have Carta’s leveling guidelines, our recent blog post Staff Engineering @ Carta shows how different two similarly leveled profiles can be.

At Carta, we make leveling decisions before we make yes/no decisions. Leveling is determined by answering, “What is the highest level we can bring this candidate in at and see them do great things?” This responsibility falls on the engineering manager and hiring committee. They consider both the structured interview feedback and the candidate’s past experience to make a leveling decision. Finally, we make a hiring decision.

To level candidates fairly and consistently, decision-makers need to be calibrated against a shared understanding of Carta’s expectations for each level. Our standardized candidate profiles make it substantially easier to do that calibration. And making calibration easy is especially important because our leveling expectations are inevitably going to evolve. Since our interviewers are data collectors and not deciders, only the engineering managers and hiring committee need to go through this calibration. We can change our leveling expectations without retraining every interviewer in the company.

Different interview performance for three different candidates. All received similar offers.

Hold ourselves accountable

Our data doesn’t just help us make better decisions for our candidates; it also informs us how to improve our interview process. We can identify which attributes are too highly correlated to provide distinct signals. Or we can find out which attributes are the least informative in our final judgment and refocus them to be more useful.

High marks for operations is required, but not sufficient, for more senior levels. The operations rubric is working as intended, but communications is not.

Peter Norvig discussed the Lake Wobegon strategy of hiring back in 2006: Only hire candidates better than your average performer, and over time, the average skill level of the organization increases. Raising the average gets harder to quantify as the organization grows. Given our structured interview data, we can monitor our decision-making and quantitatively ensure that we’re raising the bar, not lowering it.

Be the change

The framework we’ve described doesn’t make interviewing engineers easier, but it does make the process less subjective, more data-driven, and easier to change. We designed our attributes and rubrics specifically to measure a candidate’s ability to succeed and thrive as an engineer at Carta, but the principles apply to any company: Figure out what attributes matter to you, define rubrics to set a bar on proficiency, and tighten your interview questions to align with the rubrics.

At Carta, our evaluation process was defined and deployed by engineers who saw an opportunity to make a difference. We expect engineers to solve organizational problems. Carta is a place that not only values that type of initiative, but encourages it.

Curious to experience our interview process first-hand? We’re hiring! Check out https://carta.com/careers/