Why Your Technical Interview Is Broken, and How to Fix It

The tech industry is suffering from terrible technical interviews that lead to false negatives and hurt candidates. We can do better — here’s how.

I jut recently completed a month-long job search in which I went through 75 interviews, resulting in 7 offers. The technical interviews I went through could be classified as good or terrible.

The good ones were great. They were engaging, fun, and left room to interact with people and go through the thought process and tradeoff discussions together. They felt like conversations and collaborative problem solving.

The terrible ones were truly terrible. Some asked me to invest five hours into take-home projects, despite the fact I had a job as well as an extensive portfolio of open-source side projects. Others ask me to spend hours to complete technical assessments that ended up not assessing anything related to the role. Others asked me to refactor or code, but then provided only skeleton boilerplate code that left little room to actually demonstrate engineering skills and technical knowledge.

As someone who has been on both sides of the hiring aisle many times, it pains me to see such terrible interviews being inflicted upon candidates and companies alike.

It does lasting damage

Bad interviews hurt candidates, who have to be exposed to stressful situations that leave them feeling defeated and dejected, even if they would make otherwise excellent hires.

They also hurt the companies themselves by extending the time it takes for them to hire, as well as leaving them with poor reputations amongst the candidates who have to endure it.

The process is broken

Companies simply don’t know what they are hiring for. Because of this, they use the wrong or ineffective technical assessments and evaluate the wrong skills.

Know the skills you’re hiring for

Did you know there’s differences between the following?

• Coding
• Programming
• Engineering
• Architecting
• Developing

Are you using the appropriate technical interview questions to identify the appropriate skills needed for the specific role you are hiring for?

Coding

Coding is the mechanical process of entering code instructions into the computer. Familiarity with language syntax, standard libraries, etc. are the skillsets involved here.

Examples:

If you’re asking someone to write a for-loop that prints the numbers 1 through 10, you’re testing their coding skills.

(1..10).each { |i| puts "#{i} " }
# 1 2 3 4 5 6 7 8 9 10

Problems that assess coding skills often:

• Have a single method that has a deterministic result
• Have a well-defined or easily discoverable algorithm
• Have well-defined inputs
• Have well-defined outputs

The prompt might look like:

def fizzbuzz(n)
  # Print fizz if the answer is divisible by 10
  # Print buzz if the answer is divisible by 5
end

The above example has a very clear input, a very clear process, and a very clear output.

Even in cases where the algorithm is not explicitly defined, it is still relatively simple — there’s no complex calculations, forward-looking/backwards-looking logic, or gotchas.

What does this tell you about the candidate?

Candidates performance in coding gives a few clear signals:

• Are they aware of the language constructs available to them?
• Can they take a sequence of defined steps and input them into the computer?
• Can they translate the real-world steps into actual, working code?

However, there’s also some noise. Performance doesn’t necessarily indicate:

• Whether people can structure programs maintainably
• Whether people can implement business requirements
• Whether people can optimize systems
• Whether people can solve problems
• Whether people understand the technology, how to integrate with them, or the tools available to them

Coding assessments are great for filtering out people who don’t know anything about writing computer programs.

Programming

Programming is the act of creating a sequence of steps that can be translated into code to solve the problem at hand. Knowing algorithms, the various attributes of data structures, and problem analysis techniques are the name of the game here.

The solution space for a good programmer is optimal runtimes, minimal growth curves, and low memory and processor usage.

Also important to note is that there are two rather distinct types of programming problems.

Simpler programming problems like “check if this string is an anagram of another string” can be done with basic problem solving.

Advanced programming problems like “what is the minimum number of moves this knight must make before they have visited each square at least once?” requires a certain level of mathematical or algorithmic knowledge that many find difficult to derive or use in an interview setting.

Examples

Programming questions are often arbitrary problems with some sort of brute-force solution, and a hidden optimal solution.

The problems often are abstract or don’t particularly represent any real-world use-case.

The solution itself may be hidden behind a “gotcha” — a solution you would only reasonably know if you had seen it before.

Programming problems may read like below:

• Write a function that finds the length of the largest substring without repeating characters.
• What is the minimum number of moves this knight must make before they have visited each square at least once?
• Write a function to sort this list of numbers.

The medium is often a whiteboard, coding pad, or an online assessment. This is the kind of dreaded “whiteboard interview” that many developers hate but the industry embraces.

What does this tell you about the candidate?

The signal:

• Can they develop or optimize algorithms to manipulate data in an effective, efficient way?
• Can they optimize programs?

The noise:

• It does not show you whether people can structure programs maintainably.
• It does not show you whether people can implement business requirements.
• It does not show you whether people can solve engineering problems.
• It does not actually show you whether people can solve programming problems, either.

Wait — what about that last point?

It turns out the signal here may still be noise.

You see — there’s only two ways to solve these programming problems: you have to derive the algorithm on the fly, or have to know it already.

This is interesting. By studying specifically for these programming questions, you can excel at them. But, you can study them in two ways — either by truly learning and knowing how to derive the solution, or rote memorization of the solutions and a bit of luck encountering a known problem.

This assessment style specifically incentivizes candidates to lie and pretend they haven’t seen the problem before to better emphasize their problem-solving skills. At that point, is there an actual signal?

Additionally, by introducing the element of studying, this test specifically favors candidates who have this information is fresh in their minds, such as recent graduates of computer science programs. It actively disfavors people who have been in the industry gaining experience building things, or people without the time to study. It is a hidden, unspoken, albeit indirect bias against age and people with families or other commitments.

All the worse, most companies don’t actually need this kind of problem solving ability. With the exception of some larger, data-heavy, at-scale organizations like Google or Amazon, what most companies really need are people who can build features, structure systems, and make them adaptable to changing business requirements.

In these cases, such programming problems are a valueless assessment. These solutions do not translate into effective problem-solving at the product and systems level. You are not FAANG company, so stop interviewing like them.

This is also not mentioning the fact that computer scientists have spent decades of their lives originally developing these algorithms, and these whiteboard interviews often expect you to derive the solution in 30 minutes.

“We need to figure out if the candidate can code”

This is often the go-to reason when I ask companies why they do these style of interviews. What people forget when designing their technical interview questions: coding and programming are two separate skills.

They incorrectly conflate the two. Programming implies coding, but you can be a great programmer and a terrible coder, or vice versa.

Let’s say you are an algorithm developer that knows the algorithm, but you’ve never coded a single instruction in your life. In this situation, you are not a coder, but you are a programmer.

Being a terrible coder is often also be a temporary state of being and is contextual. For example, I know Java, but if you asked me to develop in C#, I will be a terrible coder until I can ramp up in the available language constructs, which may end up being a matter of a week or two.

This is where the industry is getting it wrong.

Most technical interviews leveraging Leetcode, HackerRank, CoderPad, et. al are testing algorithmic skills first and foremost. They aren’t actually testing knowledge and experience coding, engineering, or using and incorporating technologies into systems, as many believe.

These algorithmic aspects are where most people trip up in their interviews. If they can’t derive the algorithm, they can’t convert it into machine instructions, even if they could have coded it well had they known the algorithm. You can tell this is happening when the candidate seems to be writing tons of code perfectly, but can’t seem to get the write output.

It makes sense to use algorithmic questions if you are hiring people to develop algorithms like Google, but I guarantee most companies are not. I’d personally reject any code where a developer on my team tried to write their own sorting algorithm instead of calling Array.sort().

Companies using these kinds of programming tests then receive a lot of false negatives. They might feel great about having such as high bar, but it is artificial.

Maybe the candidates aren’t as strong at programming, but are fantastic engineers and coders. You’d never know, because your interview doesn’t even test for engineering skills, and the problem doesn’t give space to demonstrate coding ability until you first solve the problem of the algorithm.

Want to find out if this is the case for your candidates? Write out the solution sequence in human-readable sentences in your problem description, and see how many more candidates suddenly start passing your test with flying colors.

What I found while hiring is that people who end up being excellent engineers often do terribly in the programming tests. They hadn’t been studying how to solve arbitrary problems that have already been solved. They’re not fresh out of their university’s computer science program. They’ve been out in the field building systems. Their algorithmic problem-solving skills may have started to stagnate, because those skillsets are completely separate.

Engineering

Engineering is the process of identifying and structuring components into a system that accomplishes a goal and fits within a set of constraints within a given context.

This is the realm of data models, architectures, software designs, interaction structures, data flow management, etc. Engineers ensure your system doesn’t topple over, is secure, can flex and bend to meet changing requirements, and fulfills your business’ operational requirements.

Don’t automated online assessments evaluate this?

HackerRank and CoderPad don’t test for this. They simply can’t — at least not yet.

Engineering is part art and part science. It’s not something that can be filtered for with a multiple-choice question or demonstrated in a single function you can write in an hour.

It requires deep understanding of the business and context, analysis, and back-and-forth discussions to identify potential change points and edge cases.

A good example of a bad question

One question I saw was a multiple-choice question with a 60-second timer that went something like this:

1. If you had to model users sending messages, which would be best?
a. A User table and Message table where Message has a sender_id and receiver_id
b. A Message table where Message has the sender's name and receiver's name.
c. A Message table that has the sender_id and receiver_id, sender_name and receiver_name.
d. A Message table where the message has the name of the sender.

The problem? There is no such thing as an absolute best in engineering. There is only contextually appropriate. Any one of these answers could be a great solution depending on the constraints in the environment, which include business factors like time, cost, and available skill level. They might all even be unfit solutions for the situation at hand.

Solution A was the “correct” answer that the system wanted me to put in, but there’s many scenarios in which it might be a bad decision. What if you had a group chat? Would you want to duplicate every single message to every single person? What if you wanted to send messages to people without a user account?

Solution B can be a good solution in some cases. What if you wanted an anonymous drop-off messaging system, without any tracking? What if the sender and receiver are groups?

Solution C can result in duplicate data, but also provides some advantages. If sender name changes are common, and you want a snapshot history of it, storing the name at that moment in time with the message would allow you to fulfill that requirement. Since the data is denormalized, the resulting reads are greatly sped up if performance is a factor.

Solution D might be an acceptable solution in older systems that don’t have the capacity to track users separately from the message, or if you need to track the message in its entirety. If the system is sending/receiving a text message from a single number, and you need to track the contents of what you sent while notifying the recipient, you’d have to add the name to the message or at least some identifier.

There’s also a ton of other potential solutions, many that might involve not having tables at all! Document stores are a thing, as is peer-to-peer ephemeral messaging.

The point

The point of this is that the context can change the “correct” answer wildly, and there’s many assumptions baked into even this simple, generic question. Someone who immediately chose A as the correct answer without even considering any of these factors is likely NOT going to be the kind of engineer you’d want.

Context is everything, and there is no “best” solution.

So, how do you evaluate this skill?

You identify the signal through:

• Discussions on tradeoffs, factors, decision-making on technical projects, both past and hypothetical.
• Looking at past work, projects, and experience.
• Looking for deep understanding of interactions, structure, and fundamentals.
• Discussions about implementations, technology choices, what-ifs, etc.

This will show you whether they understand the tradeoffs and factors and structures they worked with, and whether they can apply and change their library of knowledge and tools to fit your business context.

The signal:

• Do they know how to make effective technical decisions?
• Do they know what tools are available to them to best structure the system?

The noise:

• It doesn’t show whether people can actually implement the system they talk about.
• It doesn’t show whether the system can be built or provides value in the specific situation.

Developing

Developing is the ability to build and iterate on a product or system to accomplish a business outcome.

Developing is specifically interested in value delivery. An effective developer may use the worst possible technical solution if it means maximizing the amount of value delivered.

They’ll respond to customer demand and keep the impact in mind, and ensure that the overall goals of the project are actually completed.

The noise:

• It doesn’t show whether the candidate can build systems that are easy to maintain or change in the future.
• It doesn’t show whether the candidate can build reliable, secure, safe systems.
• It doesn’t show whether the candidate can actually implement or solve more difficult technical problems.

How do you hire for this?

Testing for this involves holistic discussions of problems and solutions. Do they ask questions that keep in mind various groups of stakeholders? Do they think iteratively and incrementally? Do their solutions consider many aspects, not just technical?

Past experience here is really visible too — do they have experience building and delivering things into the hands of users? Previous startup or consultancy experience can be a part of a successful track record.

Note also that there is significant overlap in product management skills here.

Hiring

Let’s say we are looking for someone to build us a software system that allows people to log in and do stuff.

What am I looking for?

I’m not going to look for a fantastic coder. The scope of coding as it pertains to things like HackerRank is too small for what I am trying to do. I need people who can think holistically and act far beyond a known input and output for a single function. That’s engineering, not coding.

I’m not going to look for a great programmer. Problems like authentication, interfaces, dashboards, etc. have already been solved. I’m not solving novel problems, I’m structuring, re-organizing, and integrating parts to combine a set of old solutions in a new way that provides value.

I’m going to look for a software engineer with some software development skills and enough coding chops to build the idea they came up with.

What’s enough?

Coding — how much is just enough? For this situation, it would be positive answers to the following:

• Does the candidate know what variables, methods, classes, etc. are?
• Is the andidate aware of the standard and common libraries?
• Does the candidate know how to install and use dependencies?
• Is the candidate familiar with the ecosystem?

These are all things you can get from a 5 minute conversation, or a brief glance at a personal project.

Development — how much is enough?

• Does the candidate know how to track and interpret a user’s flow through the system through analytics services or interaction events?
• Does the candidate know how to refine ambiguous asks down into a small, minimum deliverable you can use to validate the solution?
• Does the candidate know how to understand the intent of the customer and mold the technology in the direction of their overall goals?

These are things you can find by asking the candidate about how they approach thinking about the product, user, and software.

Engineering — how much is enough?

• Does the candidate know how to optimize a system, and what levers are available to pull for different constraints and use cases?
• Does the candidate know how to model data in a way that can absorb change and perform effectively under multiple conditions?
• Does the candidate know how to structure your code in a way that makes things maintainable and extendable?

Programming — how much is enough?

• Does the candidate know the ways you can leverage common data structures like lists and maps?
• Does the candidate know how to read an algorithm and identify is growth rate and runtime efficiency?

You’ll notice that’s not much. Most performance related items are constrained by the architecture and system design, not by the algorithm. Even if it was, it would be easily resolved by some straightforward, basic algorithmic practices that don’t require significant programming knowledge.

Structure your assessments correctly

If you’re hiring a coder, you can use simple, well-defined problems. You can also do take-home projects, or real-time refactoring questions (provided there is enough “meat” to refactor).

If you’re hiring for a programmer, continue doing the Leetcode, HackerRank-style questions.

If you’re hiring an engineer, you start entering the realm of conversational interviews. How candidates think about factors, tradeoffs, holistic thinking, and decision-making can only be discovered by interacting with them candidate.

If you’re hiring a developer — look at their past work, or talk to them.

Hiring for the team

The final aspect of all this is that you need to hire for what the team needs.

Great teams are balanced as a whole, with people who work well together and have strengths and weaknesses overlapping. Synergy is a common buzzword in the business world, but for good reason: great teams are greater than the sum of their parts.

If your team is full of architects or engineers, you’ll likely want to balance it out with coders who can execute and crank out features.

If your team is full of coders who crank out features, you’ll want to hire an engineer to balance things out and help provide long-term stability and mature technical capabilities over time.

If your team is full of people who build things, but performance is a concern, a programmer can help come in and optimize things.

If your team is full of builders, or you need someone to bootstrap an idea and iterate on it in the interests of the business, hire a developer.

Hire right

Ultimately, effective technical interviews comes down to knowing what you’re hiring for. Without that, you can’t actually ask the questions that enable you to identify whether the candidate has the right skills.

Don’t waste time hiring for the wrong skills. Learn what you and your team need, and then structure your interview to hire for the right skills.