Your developer test isn’t working, and here’s how to fix it

At hack.summit() this week, Hampton Catlin (inventor of SASS and HAML) gave a talk about how not to hire, and focused on why things like algorithm tests are a terrible way to determine a developer’s skills.

When he scored a whopping 0% on an employer’s coding test (despite being a renowned developer), it became evident that these sorts of tests aren’t getting the point.

He said developing products isn’t about solving algorithms or math problems. Instead…

“It’s designing a puzzle…not solving it.”

And he couldn’t be more right. Imagine building Uber. You don’t code it to just get someone from Best Buy in Long Beach to the mall in Anaheim.

That’s solving a very specific problem, solving one specific route request. Solving the puzzle, so to speak.

Developing products is about having the ability to design the entire puzzle.

Being able to recognize the infinite routes someone might request and how the software will adapt to each of them.

Figuring out how to make sure the software won’t crash.

Figuring out how to make sure the same bugs don’t re-appear.

How a future developer on the team will be able to add new features easily.

Software development is for problem solvers, no doubt. But great developers are not only solvers, they’re understanders.

Great developers focus less on solving problems and more on understanding problems.

Coding tests like HackerRank, Codility, etc. are all great ways for developers to practice and even learn more about programming.

But it saddens me that so many great developers go unseen by companies using these sorts of tests as their yardstick for greatness. When the inventor of SASS and HAML goes unseen because of tests like these, it becomes obvious why this isn’t a good solution.

There’s a better way to test developers

It’s simple: we need to let developers design the puzzle.

We need to put them in front of a project that has a problem and test their ability to understand it.

We need to see if they can look deep enough into it to see what will lead us to a maintainable, scalable, efficient and as-bulletproof-as-can-be solution.

For example, if the challenge is to build a list of products with infinite scroll, the solution is a list of products that scrolls infinitely. The boundaries are clearly set to just build a list that scrolls infinitely.

But if we instead let them design the puzzle around the challenge, now they’re in the driver’s seat. Now the boundaries have been removed.

Ask them to look at the challenge and tell you what else we need to know to build a solution properly.

And it’s a good sign if they start asking, “Well, what happens when we reach the end of the list? Is there a message? Does it repeat the list?”

Now we’re looking deeper into a more adaptable solution and not just coding a list of products because that’s what was asked.

An algorithm test or something off of Codility is no different — there’s one right answer, there’s one solution to one specific problem. What are you really learning about their ability to develop products? To design the puzzle?

You’ll notice developers will be surprised and respect you when you lift the boundaries, because you’re no longer trying to ‘trick’ them to solve a problem. You’ve made it a creative, collaborative understanding of a challenge, and that’s where great developers thrive.


Great developers get more excited by understanding problems than the brief joy of solving them.

What matters is the reality of “real life” projects and how they go down.

You want developers who ask the right questions before writing a single line of code, because they have the product’s best interest in mind — maintainability, efficiency, sustainability, scalability, security, etc.

Then we see if they can code it

If they’ve proven they’re able to design the puzzle, then let them solve it as best they see fit. No boundaries on programming language, framework, libraries, etc.

They might even use some tech you’ve never seen or heard of (in a good way), and help open your eyes to their level of passion for learning and experimenting with open source.

Learn more about their real abilities by removing boundaries

You’ll learn so much about developers the second you take away these boundaries.

How do I know? Because this is part of how we test developers at my development company, X-Team.

We give applicants a project that lets them design the puzzle, that puts them in the driver’s seat, that lifts the boundaries, and reveals more than we could have ever imagined about them compared to having them just solve problems.

The result?

Few make it through the test with a decent grade, not necessarily because of the code behind the solution, but because they miss the important parts about designing the puzzle.

They forget to consider maintainability, scalability, documentation, security holes, performance/efficiency, the list goes on.

Put simply: they don’t ask the right questions (or any at all).

The developers who do it make through end up being incredibly selfless, they’re leaders of their craft, they communicate incredibly well on projects and our partners literally beg to work with them again.


There are extraordinary developers out there. But you’ll never separate the ordinary from the extraordinary by giving tests that create boundaries and limit their ability to show their true talents.

Let them successfully design the puzzle, and once you hire them, they’ll solve every challenge that comes their way. Guaranteed.

Ryan Chartrand is the CEO of X-Team, a global team of extraordinary remote developers who can join your team and start executing today.

Check out and subscribe to X-Team’s blog at

Show your support

Clapping shows how much you appreciated Ryan Chartrand’s story.