How to Create a Technical Test your Candidates will Genuinely Enjoy
A well-designed technical test makes your recruitment process more engaging for both you and your candidates.
The Technical Test phase of the recruitment process has received a lot of criticism recently. Some of this criticism is fair — a badly designed test is a waste of time for both the employer and the candidate. The most common complaints from candidates include: the tests take too long to do; they have little to no relevance to the work that would be done by the successful candidate; and the feedback from the employer is disappointing (if it even exists).
A good test can be invaluable though, so much more so than relying on alternative methods such as LeetCode. In our haste to hire someone, we mustn't forget that the recruitment process is a two-way conversation; you’re looking for the best people to add to your team, and the candidates are looking for the best employer to join up with. Take the time to be as interesting as possible and respectful to your candidates’ needs.
So how do you design a good technical test? Here are the five steps we follow at FlexMR:
1. Start with a Realistic Narrative.
Generic tests are for generic employers. If a candidate has made it to the technical test phase of your recruitment process, they’ll hopefully have some level of interest in what you do. Maybe they’re interested in your company directly, or maybe it’s the industry you operate in. Either way, you have an opportunity to leverage this interest by making the technical test more engaging and immersive.
FlexMR’s main product is our InsightHub — an online market research platform. We have a tool called Quick PollMR that we use as the basis of our Ruby on Rails technical test, as using a real tool helps to create realistic scenarios for candidates. We can test for bug fixes, new feature development, and optimisation improvements without having to fall back on contrived ‘invert a binary tree’ style exercises.
I’d be the first to admit that it’s no “Fight Club”, but it provides context to the test and offers the candidate a small insight into the work you (and hopefully they will) do.
2. Test Once, But Test Well
Decide what you want to test the candidate on before you start writing the test. Keep it high-level at this stage, and the more closely it relates to the daily tasks the successful candidate will be doing the better. For example:
* Candidate is able to write a controller with basic CRUD actions
* Candidate is able to read existing tests
* Candidate is able to write new tests
* Candidate is able to refactor an N+1 query
* Candidate is able to write basic CSS
* Candidate is able to fix a Javascript issue
* Candidate is able to use raw SQL to solve a problem
Once you have the list of testing criteria, you should fold it back into your narrative. Arrange everything into tasks, and make a note of what you’re looking for in each one. Combine criteria where appropriate — if it makes sense for a single task to cover both the CSS and Javascript criteria then do so.
Keep it simple, and keep it DRY! Ask the candidate for the absolute minimum required for you to assess their work. Asking them to write a controller and accompanying tests is fine, but having them do it more than once is pointless.
Don’t test things that the successful candidate won’t be using in their day-to-day work. If, for example, the work involves a lot of API consumption then build a test around that; if they are unlikely to be implementing FizzBuzz algorithms then don’t ask them to do it in the test.
3. Mimic your Internal Development Organisation Process
This helps you administer the test properly, and also gives the candidate a small insight into how work is managed in your work environment. At FlexMR, we use a mix of agile methods depending on the work we’re doing. For maintenance work, we use a plain KanBan board, and for more project orientated work we lean more towards using sprints; individual units of work are written as tickets on our Jira boards and assigned to or claimed by our developers.
Setting up Jira board for each candidate would be excessive and a bad use of time, but describing the expected work as discrete tasks (as you would with Jira tickets) is a great idea. It shows the candidate see exactly what is expected of them and guides them in organising their submission into sensible commits and/or branches.
An example ticket might be:
Prevent Changes to Poll Options Once LiveProblem Summary:Currently, the answer options on a Poll can be changed and deleted after the Poll has gone live. New options can also be added. This has a negative impact on data integrity and reduces the reliability of the gathered research data.Acceptance Criteria:Once a Poll is live:
* The Poll record should not be modifiable
* The Poll options should not be modifiable
* Poll options should not be removable
* New Poll options should not be creatable
This makes it clear to the candidate what the problem is and gives context as to why it’s a problem. It does not say how to solve the problem — that’s the point of the test, after all — but it shows what we’re looking for.
One thing to be mindful of is how you order the tasks. We should work to try to provide tasks in an order that is both sensible to follow and allows them to see their progress well: start with the quicker tasks and build up the complexity as you go along. A CSS change is a suitable opening task, but dipping into raw SQL to optimise a table join should be towards the end.
4. Don’t Waste Time
Your time is valuable, and so is the candidate’s. Technical tests take a lot of time to complete compared to some alternatives, so it’s important that this time is spent efficiently and respectfully. This is easier than it sounds and makes a big difference.
Once you’ve decided the tasks that the candidate will do, you should provide a starting point for them. As an example, for our Ruby on Rails technical test, we provide them with a git repo that has everything set up for them to get started on the test. All they should need to do is a bundle install. Everything else they need to have in place for them to complete the test is provided for them. The only things missing (and so the only contents of the eventual Pull Request they’ll submit) are the tasks the candidate will complete.
In this starting point, it would be a good idea to include seed data and a test suite too; this works to demonstrate the standard of work that you expect to see from the candidate. To make the test as close to the real daily tasks as possible, use the same third-party libraries in the test project that you use in a real project. For example, if you’re hiring a Rails developer then there’s no value in using Sinatra in your test.
The candidate should be clear about what they need to do at every stage. They are likely already feeling a bit of pressure, so it would be in everyone’s best interest for the employer to provide a starting point where the candidate can jump straight in and demonstrate their skills rather than have them concern themselves with writing generic boilerplate code.
Throughout the test, make sure to provide a way for them to contact you with questions. Even the most well-written tasks can be interpreted in different ways, I’d rather spend time helping a candidate understand the task before they get started then find out later.
Give plenty of time for the candidate to complete the test; we find that a week is usually sufficient but bear in mind that people have different commitments. They may still be working, have child-care or other family commitments, or they may have other technical tests that they are working through.
Keep the overall time to complete the test low. 2 hours is a good target and should be enough if your tasks and starting point are well written, but you could allow for up to 3 hours depending on the candidate’s prior experience. Any more than that and it might mean you need to rethink your test; consider questions such as: is it the bare minimum that you require? If not, what can be stripped out?
It’s also a good idea to take the test yourself, and ask your colleagues to do it too. This is the best way of finding out how well your test works. If it takes you 4 hours to complete the test you wrote, there’s no way a candidate will do it in 2–3 hours. Getting a colleague’s perspective is really helpful too, as they’ll give you a better idea of how long it’ll take a candidate to complete the test, and you’ll also get the benefit of having a second pair of eyes looking out for any pain points or repetitive sections.
Once you’ve received the candidate’s submission, don’t delay in reviewing it. Don’t rush doing the review, but don’t put off starting it either. Likewise, once you’ve done the review then arrange any follow-up tasks as soon as possible; a quick turnaround is beneficial to the company and will be appreciated by the candidate.
5. Finally, Give Meaningful Feedback
The most common complaint I’ve seen about technical tests is when the candidate does not receive any feedback about their submission. It’s a fair complaint — if somebody invests a few hours (or more) into a programming challenge at your request, then they deserve to hear back from you.
When we receive a test submission from a candidate we give it a full code review. I’ve previously written about how we do code reviews at FlexMR — they’re quite involved and highlight the positives as much as the negatives. Once this is done, we invite the candidate for a follow-up video call (where possible) so we can talk through their submission; this gives both us and them the opportunity to discuss any interesting approaches they have taken or any areas they may have struggled with and deserve more attention. Where this isn’t possible, we pass our feedback to them via the recruiter.
We’ve seen some great success with administering our technical tests this way, and have hired some fantastic people to our development team. Not all of the candidates make it through this round, but by acknowledging their efforts and being respectful of their time we’ve shared a positive experience with them and, hopefully, we’ve both learned things along the way.
