Anatomy of a screener: recruiting participants for your research

Editor’s Note: Mirjana Risek is a Senior Service Designer, and the practice lead with the User Research Lab in the Ontario Digital Service.

The Lab helps teams across the Ontario Public Service bring end-user perspectives into their design and development efforts and has been instrumental in supporting the Ontario Digital Service build key digital tools to help stop the spread of COVID-19, like the COVID Alert. If you haven’t already done so, please consider visiting the Apple App Store or the Google Play store to download COVID Alert today.

Have you downloaded COVID Alert yet?

“We’d like to test with our users, but how do we know we have the right people?”

Most of the research we do at the Ontario User Research Lab is qualitative, which means we use direct observation and interviews to learn about user behaviours and attitudes. This also means we rely on small samples of people, usually between five to 12 per identified user group.

A small sample group can provide valuable feedback, but it will be the most accurate when your participants represent your intended users as closely as possible. To do this, we use a research tool called a participant screener.

What is a screener?

A screener is a questionnaire that helps you find and gather the people who are best suited to help you answer your research questions — those who are most like your end users. If you don’t screen, your findings might be diluted with input from people who are outside your target group, and whose feedback won’t be as relevant or valuable to you.

At the Ontario User Research Lab, our screeners usually take the form of an online survey, but a screener can be adapted for other formats, like a phone script, for example.

Typically, there are four types of questions in a screener:

1. Qualifying questions are about finding the right people to include in your research.
2. Behavioural diversity questions help you find a range of people within your qualifying criteria.
3. Demographic inclusion questions help you include people along dimensions like age, gender, regionality, and many others.
4. Scheduling and contact details are about communicating with selected respondents to arrange your sessions.

Let’s look at these in more detail as we go through the various parts of a screener.

Parts of a screener, as they might appear on a mobile device.

Building your screener

Before jumping into your questions, set some context and introduce your screener. Tell your respondents what it’s for and what they can expect.

Try to cover things like:

• What is this questionnaire for?
• How long will it take to complete?
• What will happen to my answers?
• How will I know if I’ve been selected?
• Who can I contact if I have questions?

1. Qualifying questions

These are usually specific to your product or service, and will directly reflect your research goals for a particular study.

Ask your qualifying questions at the start of your screener, so you don’t burden your respondents with answering a lot of questions before letting them know they don’t match your study profile.

For example, if your study is about the experience of renewing a driver’s licence, you will want to hear from people who use the service — those who have an Ontario driver’s licence — and not from those who don’t.

You might ask:

Which of the following government service cards do you have (select all that apply):

• Ontario health card (with or without photo)
• Ontario photo ID card
• Ontario driver’s licence
• Driver licence from another province or country
• None of the above

(include ONLY those who select Ontario driver’s licence)

You may be wondering why the question above includes a bunch of irrelevant options. Wouldn’t it be simpler to ask it more directly?

For this, you might ask:

Do you have an Ontario driver’s licence?

• Yes
• No

(exclude those who say ‘No’)

Simpler, yes — but it may be less effective.

By adding a few different options, you can make your screening criteria slightly less obvious, and get more reliable responses. This technique can help improve your screening, so you don’t include people who are eager to help, or want to participate (FOMO!), but who may not be the most suitable for your study.

You will also want to exclude anyone who develops, makes decisions about, or is responsible for the renewal service — ideally you have a plan to capture their needs in a different way.

For this, you might ask:

Do you work for the Ontario Public Service (as a permanent or contract employee in a ministry, secretariat or office, not including an agency)?

• Yes
• No

(exclude those who say ‘Yes’)

And you probably want to include a question about age:

How old are you?*

• Under 16
• 16 to 29
• 30 to 45
• 46 to 59
• 60 to 75
• Over 75

(exclude those under 16)

This looks like a demographic question, so why are we talking about it here?

Since we need to know if the respondent is a minor (a disqualifier for most of our studies*), this question serves as both a qualifier and a demographics question, so we usually put it near the beginning of our screeners.

*In the rare case where we want to talk to younger Ontarians specifically, we need to know their ages because consent and research protocols are different for minors than for those 16 and up.

2. Behavioural diversity questions

These may also be specific to your service, and may be drawn directly from your research goals, but will help you find a range of people within your qualified group.

Using the driver’s licence example, you might want to hear from seasoned drivers who have renewed multiple times, as well as from those who are renewing for the first time. You might also be interested in those who use the online service and those who don’t.

So you might ask questions like:

How long have you had an Ontario driver’s licence?

• Fewer than 5 years
• 5 to 10 years
• 11 to 15 years
• 16 to 20 years
• More than 20 years

(include a range)

And you may want to know:

How have you renewed your licence? (select all that apply)

• In person
• By mail
• Online
• I have never renewed my licence

(include a range)

3. Demographic inclusion questions

These are probably entirely unrelated to your service, in that your service may be available to everyone.

But these questions are important to your research so that you can definitively say who did (and didn’t) participate in your study. You will want to describe the breadth of your research, and if you don’t ask, you can’t know. You won’t be able to see if there are any gaps in your research.

Some of these questions can seem sensitive and may lead people to wonder why you are asking them. We’ve found that by following a couple of key guidelines, we can ask them politely and get a pretty good response rate:

• Group these questions together near the end of your screener
• Preface them with an explanation for why they are being asked. Something like:

“The Ontario Government is committed to serving all the people of Ontario. The following questions help us build a better understanding of who participates in our research.”

• Make them very obviously optional, so respondents know they can skip them.*

*Except age. As noted above, we have to ask people how old they are to know whether they are suitable to participate or how to properly get their consent, so we make this question mandatory and ask it up front.

4. Scheduling and contact details

These logistical questions allow you to follow up with selected respondents to arrange your research sessions.

It’s a good practice to ask only what you plan to use. Our screeners typically ask for a name (without separating first name from last name since that’s not going to be important), and an email address so we can politely communicate with the person. We also offer the option of adding a phone number if our respondent prefers to be contacted by phone.

If your study dates are already set, as they usually are when you are ready to start screening, you can include a table of options for people to indicate dates and times they are available.

This can help you with the next step, which is scheduling your sessions.

What happens next?

At the end of your screener, make sure you let respondents know what to expect next. We usually include a phrase to let them know that if they are selected, they will be contacted for scheduling.

Politely disqualifying

Along the way, if your respondent answers a question that triggers a disqualification, make sure you let them know what’s happened. But depending on how you have set up your screener, you may want to avoid being too specific. (You don’t want to encourage people to go back and answer the questions less accurately so they can participate.)

A ‘disqualify message’ we often use is something like:

“Thanks for your interest in our research. Based on your answers, you don’t match the specific profile we’re looking for in this study.”

And here again, you may want to include an email address where people can send any questions they might have about the research.

See it in action

To see how these elements all tie together, have a look at a preview version of a recent screener we used to find participants for a number of COVID-related studies. Feel free to play around with this one, as it doesn’t feed into the respondent pool, but does show how we use survey logic to streamline the questions.

Participant screener for COVID-19 related user research

Want to learn more?

For more on screening, check out this great post by Nielsen Norman Group on How to use Screening Questions to Select the Right Participants for User Research.