In-Product Intercepts for User Research Recruitment

A story about recruitment methods that worked 3x better for us.

Recruiting interview subjects for a research study can be hard. You spend countless hours crafting recruitment campaigns and eligibility surveys, send 100s if not 1,000s of emails, and yet still have trouble finding candidates. This is a story about a recent project, and some recruitment methods and tricks we tried that worked 3x better for us.

First, a little bit of context

As Related Works, we help a wide spectrum of companies more deeply understand their customers and build better products. A central part of our work is conducting quantitative and qualitative research, which often includes a series of research studies with specific customer cohorts.

Earlier this year, we had the opportunity to participate in VX2020, an initiative by Higher Ground Labs to provide quick, two-week research sprints to companies that build critical tools used by progressive organizations and campaigns. As part of that initiative, we got to work closely with Mobilize, a white-label events platform that organizations and campaigns use to publish and manage their events.

We did a number of sprints with them, allowing us to try a variety of different recruitment methods with drastically different results. Our hope is that by sharing our experience, we can help similar product teams more effectively recruit high-quality research participants as well.

Sprint 1: Email recruitment

Working with Mobilize, we decided that our guiding research question was: “How can we improve the first-time volunteer experience?”. We knew anecdotally that Mobilize is just a single touchpoint in a long volunteer journey, so we wanted to shed some light on this critical part of the experience by speaking to 5–7 first-time volunteers in 1:1 interviews.

Once we knew roughly who we wanted to talk to, we then needed to define these three key inputs:

1. What is our specific candidate criteria?
2. Are we going to offer an incentive for their time?
3. What recruitment method should we use?

When thinking about our candidate criteria, we needed to identify what key behavioral and demographic attributes were important. Considering this was going to be the first round of research done with first-time volunteers, we wanted to make sure we were talking to a diverse and representative group of Mobilize’s user base. We ended up with this set of requirements:

Mobilize Experience:

1. Recently completed their first volunteer event using Mobilize, ideally within the past week
2. Diverse range of organizations or campaigns
3. Diverse mix of event types, such as canvassing, debate watch parties, phone banking, etc.
4. Diverse mix of device usage, slightly weighted towards iPhones

General Demographics:

1. Geographically diverse, and from a mix of urban and suburban areas
2. Diverse mix of ages, slightly weighted towards older participants
3. Diverse mix of genders, slightly weighted towards females
4. Diverse mix of ethnicities

If you’re new to defining a set of candidate criteria, this article does a great job explaining what to consider and look out for.

Next on the list was figuring out our research incentive. Typically, it’s a good idea to compensate research participants in some way for their time, but how much and in what form can vary depending on your budget and the level of time and effort you’re asking of them. We decided to offer $50 Amazon Gift cards for about an hour long conversation over video, which is what we typically do for similar research studies.

We then needed to figure out what recruitment method was going to be best based on our candidate criteria, sprint timeline, and any existing lists of potential research candidates we could pull from. We sat down with the Mobilize team and reviewed some commonly used recruitment methods to see which one made the most sense for us:

1. Doing some more guerilla-style and in-person recruitment by going to some of these upcoming volunteer events that were hosted on Mobilize, by reaching out to potential candidates from our own network, or even through Craigslist or relevant Facebook groups.
2. Adding an in-product intercept to Mobilize after someone has registered or attended an event.
3. Directly emailing individuals that signed up for an event for the first time on Mobilize.
4. Recruiting from an existing research database such as userinterviews.com, ethn.io, or usertesting.com.

Considering we were on a very short timeline — we needed to have interviews scheduled and conducted within 4–5 days — doing an in-product intercept and more of a guerilla-style approach was not going to be feasible. We went with the quick-and-easy route: Mobilize helped us compile a list of individuals that had recently attended events, and we emailed them. Since we had limited demographic data, we asked users to fill out a screener survey in order to confirm we were going to be talking to a diverse group of individuals who met our criteria.

Based on our experience doing similar research in the past, we expected around a 5–7% response rate, but only around 2% of users responded in this case. One hurdle we hadn’t anticipated is that since Mobilize is a white-label product, end users generally aren’t aware of Mobilize. We learned from the users we eventually spoke to that some were even suspicious that our emails might be malicious. We suspect this lack of brand-awareness played a big role in our low response-rates, and imagine that this same hurdle may apply across a wide spectrum of white-label products.

A couple different organization event pages that all are using Mobilize.

We were eventually able to recruit enough candidates, but ended up having to email many more users than we had anticipated.

Sprint 2: Experimenting with in-product intercepts

Thankfully, we were able to conduct a second research sprint a month or so later, this time with a little more time to prepare. We still wanted to talk to first-time volunteers, and we knew that directly emailing individuals was sub-optimal given the hurdles around brand awareness we experienced the first time. So we decided to try something different this time.

We reviewed our list of possible recruitment methods again, and in-product intercepts seemed like it could be a more promising option this time around. After someone has attended an event, Mobilize will send a quick follow-up text asking how the event went. We wanted to see if we could leverage this interaction and add an additional follow-up text for those that responded and had attended their first event, asking if they’d be interested in participating in a round of research:

The Mobilize team was amazing and was able to quickly add this intercept for eligible candidates. Within a couple of days of turning it on, we got roughly a 7% response rate, over triple the response rate compared to our first sprint. In addition, since we were able to catch people within a day of attending their event, the quality of the interviews improved, as people were more likely to remember important details since it was still fresh in their minds.

Conclusion

Figuring out which recruitment method to use given your research goals and constraints can be challenging, especially when you’re trying to reach individuals with limited brand-awareness. In our experience, in-product intercepts are an effective method because you’re catching the user while they are already engaged and active within your product.

If implementing your own custom intercept is not possible, there are a variety of intercept services you can easily implement, such as Ethnio’s Intercept Tool. The Nielsen Norman Group wrote an amazing step-by-step overview of how they used it to conduct research on their site. We’ve found intercepts to be an essential recruitment tool in our research, and we hope you find them useful as well.

Many thanks to everyone at VX2020, Higher Ground Labs, and the Mobilize Team for the opportunity to work together, and to Giovanni Fernandez-Kincade (my partner in work, life, etc), Mollie Ruskin, and Peter Martinazzi for taking the time to read this post and give thoughtful feedback to make it better. 💖