Make Your Research Stand Out with High-Quality Screening: Recruiting Tips & Tricks
At Marketade, researchers roll up their sleeves and own projects from start to finish — kickoffs, recruiting, session design and moderation, workshop facilitation, report writing — and without a doubt, recruiting is the phase that runs the highest risk of extending a project’s timeline if it becomes challenging.
To raise the stakes even higher, recruiting that doesn’t match a client’s target customer can immediately devalue a research session from the client’s perspective. Since we immerse our clients in research sessions, we can’t run the risk of hearing them say, “That’s not our audience,” or, “We aren’t interested in this person’s feedback.”
We need to get the right person for every session, every time — and we do! Over the last few years, I’ve had the opportunity to recruit participants and users for countless research projects, and I’m going to share with you some guidelines we use for ensuring high quality recruiting across the board for our qualitative sessions.
(Note: Not all recruiting is a doozy. Some recruiting doesn’t run the risk of extending project timelines, or ever require a phone screener. And in some cases, “close enough” recruiting can truly provide valuable insights for the client — this isn’t about those times! You’re the best judge of whether this approach to recruiting is a good fit for your project. That said, I find that the mindset this playbook puts me into helps guarantee high quality recruiting on even simple projects.)
The Rule: Why Our Recruiting Stands Out
Some of our clients come to us having had bad experiences with low-quality recruiting services for prior usability research. These services rely on what we call “close enough” recruiting tactics, and might take someone’s word on a screener survey without calling to discuss and verify.
When I’m recruiting for a study, I approach conversations with potential participants armed with hidden skepticism (verify!), open-ended questions (let them say it in their own words), lots of intuition (this comes with experience), and the mindset that an online screener survey is just the start of the recruiting process, not the bulk of it.
I’ve even created a rule for determining whether a participant is a good fit for a study. I call it The Rule. Creative, right? Here’s The Rule:
The situation we are going to put them in during usability testing is so realistic for them that it should be fairly effortless to get them to recite your recruiting criteria to you (without prompting) during a phone screening call.
Read it, mull it over — I’ll talk more about The Rule later.
The Client is the Expert: Learn From Them!
Sometimes, a client trusts you to figure out who the best participants are for a project — other times, they have already have a specific target audience in mind. Either way, give the client an opportunity early on in the engagement to share their expertise, and collaborate on a loose recruiting plan together. I try to do this on a kickoff call at the very start of the project.
I think of the kickoff call as the time to get recruiting set, so I can get going — in the best case, I’d love to be able to hop right off the call and draft a screener survey. I usually plan at least 15 minutes for a recruiting-related discussion, which I prompt with a slide titled Recruiting. I never leave it blank — I try to fill it in with anything I might already know about their target user, and treat it like a mutable document we’ll then collaborate on together.
I treat the recruiting discussion like a miniature collaborative workshop, that moves along fueled by their knowledge of their customers and my knowledge of research sessions and recruiting. Ask open ended questions like:
- Are there any demographic considerations to keep in mind?
- What are your customers’ goals and desires?
- How will I know if I have the right person on the phone? What about the wrong person? What types of things might they say that would tip me off?
- What types of devices will we be using for the sessions?
- Are there any voices I’m missing here?
- Is there anything else I need to know about who these customers are?
A Good Screener Survey is Just the Beginning
I like to think of a screener survey as the equivalent of asking someone, “How are you?” and having them reply “Good, how are you?” It doesn’t mean they’re actually doing well — it just means they’re jumping through the hoops of the conversation and willing to speak further. A screener survey can tip you off that someone might be a good person to continue speaking with — that’s all.
As a guideline, I use screener surveys for ruling out, not ruling in.
For example, imagine you only want to talk to customers of a particular rental insurance company — that’s a great thing to use a screener survey to filter for. You can ask more specific questions if you want (“Have you ever had a rental claim? When?”), but don’t necessarily trust their answers, and you should expect to ask it all over again on a phone screener to verify and learn more.
Some quick screener survey best practices I try to follow:
- Keep it to 5 questions or less, preferably all multiple choice.
- Always make sure you have options for “Does not apply,” or “I don’t …”
- Rephrase, hide, or remove dependently phrased questions. (Turn “If you have renter’s insurance, how long have you had it for?” into “How long have you had renter’s insurance?” with an option for “I don’t have renter’s insurance.”)
- As best you can, conceal the “right” answers amidst other reasonable answers that don’t seem like they’ll be disqualifiers.
- Make general questions specific. Rather than “Do you use Facebook?” Ask something like, “How many times did you check Facebook today?”
The Phone Screen: The Only Thing That Matters
The phone screen is the only thing that matters.
So you have a big list of people who have filled out your screener survey. Now you can start making calls to people who seem good so far. I suggest using Airtable to keep track of recruits, or Google Sheets works too.
Create a big master spreadsheet, and add a notes column where you can take quick notes while you chat with the potential participant. Use color coding to track where they are in the pipeline (Left Message, Not Interested, Bad Fit, Scheduled, Paid, etc.) and keep track of each way you contact them and when— emails, voicemails, texts, etc.
Avoid telemarketer-type reactions by reminding them they initiated the contact. The first thing I say on a call is, “Hi, I’m Sonya — I’m calling about the survey you filled out on UserInterviews.” This reminds them that they took the action that prompted my call, and helps alleviate the possibility of them thinking I’m a salesperson making a cold call. Often they think for a moment, remember the survey, and then respond with a marked increase in warmth/familiarity from their initial skeptical “Hello?” when they saw the unknown number.
Snag the scammers by holding off on details. Often people that fill these out full time will immediately try to get a hint (or ask directly) about which survey you’re calling in regards to, because they’re trying to remember which (fake) set of information to provide you with. As soon as possible, try to ask them a basic question from their survey that you already know the answer to (“Which do you maintain: renter’s or homeowner’s insurance? Can you remind me who that’s with?”). And, remember to stay neutral throughout the phone screen so you don’t tip them off about what you are or aren’t looking for.
Call absolutely everyone that might be a fit. Your screener survey should filter out the people that really don’t matter, and then tell you who to start with. But call everyone, or as close to everyone as you can handle. You’ll be grateful later if there’s a no-show or a low call-back rate. Bulk calling sessions pay off, because leaving voicemails means they virtually come to you! Say to yourself, “I am going to make 30 calls today.” Do whatever you have to do to get motivated about getting these people on the phone, because I think once they’re actually on the phone, it’s fun. Imagine these are all very cool people at a party you are getting to meet.
Re-ask everything from the survey. Their answers should be right in front of you, and it’ll just take a second. It’s way better than getting all the way to a session and realizing they have the wrong brand of insurance, just because you didn’t verify.
Use what I call the “kind of dumb ellipses trick” to get recruits to elaborate. “And you filed the claim …..?” This tells them that this isn’t super formal, you have time, and you want the whole story from them. It’s simple, and it gets people talking!
Back to The Rule
Remember The Rule, from earlier? Here it is again:
The situation we are going to put them in during usability testing is so realistic for them that it should be fairly effortless to get them to recite your recruiting criteria to you (without prompting) during a phone screening call.
So, imagine you’re looking to speak with people who have recently gotten into a car accident, and then filed their insurance claim online — using a smartphone. Perhaps you used the screener survey to filter down and find people who have filed claims recently, and people who own a smartphone.
Here’s what a BAD phone screener might look like:
“Have you filed a claim in the last two months?”
“Yes”
“What happened?”
“I got rear ended.”
“Did you do it online?”
“Yes.”
“Were you using a smartphone?”
“Yes.”
This is bad for a lot of reasons. The questions are yes/no (rather than open-ended), very leading, and you don’t get a sense of the recruit’s actual experience. I would never approve a candidate based on a phone conversation like this — it’s not adding any extra value to the screener survey, it’s just an extension of it. And, most of all, it doesn’t follow The Rule — the participant didn’t speak your recruiting criteria back to you in their own words.
Here’s what a great phone screener should look like for this scenario:
“When is the last time you filed a claim?”
“About two months ago.”
“Tell me about the process.”
“I had just gotten rear ended.”
“And you filed the claim …..?” (Sometimes you have to prompt them, encourage them to open up. Remember the ellipses trick?)
“Yeah, I called in right then to file the claim, and then they told me I had to do it online, so they texted me the link to start it and I kinda started it then and then I went home and I went through the whole process online. It was pretty easy, it went ok. I’m still kind of irritated because I don’t think I was compensated as much as I should have been, but the process of it was honestly really easy.”
“Ok and you were doing this on, like … Smartphone? Computer?”
“Oh, my phone. I use my phone for everything.”
Boom! That was wild! That person just spoke your recruiting criteria back to you, without you even trying. “About two months ago … I went through the process online … on my phone. I use my phone for everything.” When that happens, you followed The Rule and found a great candidate whose insight is almost guaranteed to be valuable for the project.
Gut Check
If someone’s phone screen passes “The Rule” test, the last thing I do before approving someone for a session is a quick gut check.
- Do you think this person was honest?
- Could you see this person being a customer or user of this service?
- Will this person be a good fit for the device we’re asking them to use?
- Is the scenario we’re testing going to sound familiar and comfortable to this person?
- Did the person identify themselves as fitting the recruiting criteria?
- Did you get a good feeling about talking further to this person overall?
I’m proud of the recruiting I do as a result of following this playbook. It took me a while to develop my recruiting intuition, but ultimately it’s one of my favorite parts of being a user experience researcher. If you have any questions, please feel free to reach out! I always love talking recruiting and research.
