Consider Context When Designing Your Work Sample

Developers spend much of their paid time either writing code, or thinking about writing code.

As a result, having prospective hires actually write code as part of the interview process is a pretty good idea.

(In fact, this idea is #11 on the Joel Test, which is a great baseline set of rules to follow to be a good place for developers to work at. Lots of other people have written a lot of great stuff on this subject.)

Once you’ve agreed that having candidates write code as part of their interview is a good idea, the fun begins. Countless questions arise, such as:

• Why are we doing this?
• How do we test for the right things?
• What tools should the candidate be allowed to use?
• When should they complete the work sample?
• How long should it take?
• How do we actually know one candidate is better than another?

Each one of these could be a blog post in itself. Let’s focus a bit and think about how the context of a work sample should affect the design.

Consider two work sample exercises in two different places in a recruiting pipeline.

One work sample sits at the very beginning — before the candidate even interacts with a human.

The other sits at the very end, after hours of screens through talking to different interviews, much resume review and talking about all sorts of other things.

These exercises should obviously be designed very differently, in order to be effective.

An early stage work sample should:

Be quick to score.

At this point in the pipeline, you may have many candidates. Stay away from open ended, subjective problems at this stage.

Fizzbuzz, although simple, has an amazingly high failure rate when given as an initial screen.

Some ways to make this quick would be to use an off-the-shelf platform like Hackerrank, or provide stubs with failing test cases that the candidate must then complete. Scoring can simply be the number of passing tests (or a boolean pass/fail).

Be quick to complete.

The candidate doesn’t yet have a fully formed idea of your organization at this point.

Asking them to build a full-on CRUD app is silly, and unfair. (It’s probably still silly for any other stage as well.)

If you want to screen people for “passion,” screen them based on talking to them or reading their cover letter. Screening them on their desire to give up 3–4 hours (or even days) to write code for a company they barely know about is wrong, and pretty useless.

Have a clearly defined time limit.

This accomplishes two things.

It prevents unqualified candidates from spending too much time failing. If it takes you 2 hours to complete Fizzbuzz, you should automatically fail. Don’t prolong their failing.

It also makes the candidate’s decision to proceed or not, easier. If a candidate is currently interviewing at 5 places, it’s easy to make 5 blocks of one hour. It’s much harder to make time for 5 arbitrary-length coding exercises.

Be asynchronously scheduled.

Neither side is heavily invested at this point.

Allow the candidate to pick their own time slot, and schedule an email to be sent at that time.

Don’t force them to go into your office, pair with you on Google Hangouts during the work day.

It’s bad for both of you. (If you ignore this advice, I wish you luck fitting 100+ Fibonacci sequence generator exercises into your calendar.)

A later stage work sample should:

Be difficult enough for candidates to distinguish themselves from one another.

In the extreme case, where a work sample is the final step of interviewing a candidate, you have no more criteria to rank candidates based on.

Your problem of choice needs to give the candidates space to perform better than one another.

Your scoring criteria (a whole other blog post in itself,) could include polish items like “fixed this label alignment,” or “added useful user-facing error messages” as criteria, that some candidates would miss.

Still have a well defined goal.

This seems contradictory to the previous point, but it’s not.

“This is a repo, we’ve written up some issues, fix what you can,” is okay.

“Design a system to take in data from these different sources” is still okay.

“Make an interactive website about doge” and “impress us” are not.

Although you don’t want to give away your entire scoring criteria, it should be pretty clear what the goal of the exercise is. Be clear, so that someone who completes the work sample can say, “I killed it, and I added these other things for a little bit extra.”

Still have a clearly defined time limit.

This time limit should probably be longer than your early work sample (since the problem is harder,) but the same rationale applies.

You don’t want to introduce bias against people with jobs (less time/energy to write extra code) or encourage other self-destructive behaviour.

Do you rank the guy who made a 4 star app in a day over the guy who made a 5 star app in three days?

What if someone spends a week making an awesome app to show how great they are, but they still don’t quite pass your scorecard?

Should you feel bad? (Yes, probably) Bad enough to hire them when they’re not the best fit? (possibly)

Setting a clear time limit means you don’t have to deal with these uncomfortable questions.

Be somewhat reflective of the actual work the candidate is doing.

The work sample should probably not be completely based on actual problems you are dealing with. (There are many opinions on this, and there are some good ones for using actual problems and paying people for their time. These opinions are mostly out of scope. I don’t make decisions about paying people, I just score these things and write blogs about what I think later.)

However, this would be a great time to see how the candidate performs with your framework of choice. Maybe you’re not 100% on React yet, but you’re hiring someone to get you there, and a React-based work sample would be a good way to tell if they would be good at this.

Maybe all you do a lot of customization and re-theming, so it’d be a good time to test CSS debugging ability and design sense.

Make sure the work fits, because honestly, what good does it do anyone if you have your prospective Android developers fix a webpage? It just looks like trolling.

Have some room for communication.

The larger scope means that even if you write the perfect spec, your candidate is likely to have questions. Unfortunately, this makes scheduling more of a challenge than before.

Hopefully, this sample is actually late enough in the process that less scheduling is involved. (If not, make previous stages harder.)

You definitely don’t want a situation where your star candidate reads your spec on Saturday and has no idea what you were thinking. You have more valuable things to test for, than whether or not your candidate has psychic powers.

This is also a good time to remember that interviewing is a two way street. Now, you can show that you are a well adjusted interviewer, and can be pleasant and not-grumpy when people ask you questions. :)

Allow developers to use the tools that they are familiar with.

You should allow your candidates to use their own tools, such as editor, and computer, when possible.

I’ve had a few pretty strong arguments about this in the past (mostly when the work sample was part of a multi stage onsite interview) and I still conclude that unless your workplace is filled with kleptomaniacs, seeing how candidates deal with the stress of having their MacBook Pro 15" replaced with your IT department’s beater Windows laptop is not actually a useful hiring criteria.

And once again — interviewing is a two way street. How would you feel about taking an offer if you spent most of your time slot remembering how to use Windows because they couldn’t have the courtesy to tell you to bring your own computer, and leave you a card with the wifi password?

TLDR: actually think about context and what you’re trying to do. Also, be fair to each other.

Where we go from here:

I’m not sure yet. I have a lot of assorted opinions on tech, mostly around the soft side of being a developer.

This post in particular was triggered by a recent blitz to interview 40+ intern work samples in a day. (It was a standard exercise given to all interns applying through a certain program.)

We’ll see how well I stick to my goal of regular writing practice. As a preview, future subjects might be:

• things I learned about mentoring
• optimizing for the long term, or why you should probably leave your comfy job
• code review isn’t about correctness