When incentives aren’t enough

Have you ever wondered how to reduce the number of users that fail to show up to research sessions? Have you ever thought that increasing the incentive is the answer? Here’s how the FanDuel UX Research team learned that incentives aren’t always the key.

In 2019, we started to log operational metrics of our qualitative research sessions and it opened our eyes to the truth about incentivising research and how it relates to user attendance rate.

While we’ve always logged research insights, we didn’t track the logistics of each research session. Things like; number of users contacted, number of slots to fill, number of users who attended, etc. We decided to gather the data to try and spot patterns, and opportunities we may have to reduce the number of users who book in for research sessions but then don’t attend, which is a major pain point for us.

As a team, we already recruit more users than we need for research sessions to give us some allowance for no shows but sometimes that still isn’t enough. We wanted to be able to try different incentive levels and then measure the impact they had to understand if it was meaningful.

The data points we tracked in the log were:

• Product the research was carried out on
• Research date
• Name of the study
• Number of users contacted
• Target number of users we wanted to book
• Number of users attended
• Number of users engaged with the sessions
• Attendance %
• Engagement %
• Date the recruitment email was sent to user
• Incentive offered
• Session length
• If a screener survey was used
• Number of FanDuellers who attended
• If the research was remote or in person
• Notes

We filled out the log with all our research sessions over a 6 month period and reviewed the data during a two day research team workshop in January .

What we learned was eye-opening.

On average our no show rate was 38%

Which means for every 3 people we recruit for research only 2 are likely to attend. That’s not a good rate, and came as a surprise. We knew we had an issue but looking at a cold, hard number was an eye-opener. If I had to guess before we started the log I would have said around 10%.

There was no correlation we could find between incentive and no show rate

This was the biggest surprise. We had assumed that this would be a key lever for us to improve the attendance rate. We tried experimenting with incentive type and amount but we couldn’t find any correlation to no show rate in the data. There is a caveat here in that we pay double for in-person research than we do for remote since people have to spend time and money travelling. We feel that this is a fair way to say thanks for the extra effort. While we experimented with the type and amount we always paid around double for in-person sessions.

Our no show % rose sharply in August

This makes a lot of sense as August is the least active month in the sporting calendar. Less sports in season means fewer customers on our products and less engagement with the brand. If FanDuel is not front of mind for users it could be easy to forget they have a session booked. While this is a logical conclusion, we wouldn’t have been able to say for sure without having the data points to confirm.

The day of the week the research took place was more impactful than the incentive

User’s were more likely to show up to research sessions that were on Thursdays or Fridays. Which also makes sense as people may be more likely to work from home or not be working towards the end of the week which can make it easier to attend.

When the recruitment email was sent also had a significant impact on No Shows

The timing is again key. When recruitment emails were sent out in the same week that the research sessions took place, no show rate was below average. We’d previously tried to give users a week’s notice.

Screener surveys help too

Having a screener survey that allows users to express interest in taking part in future research sessions and then recruiting from that pool of users means more show up. The log showed us that every research project we had 100% attendance for had a screener survey.

Some of the learnings were easy to act upon. We now try, as much as possible, to schedule research on Thursdays and Fridays and we send the recruitment emails out to users on Mondays. We now know as a team it’s going to be difficult to get the insights we need in August so we’ve updated our research plans to ensure we are getting key learnings before then.

We took the rest of the insights from the log into a compressed design sprint (similar structure to a standard design sprint however compressed into one day) to create new strategies and material for acting on the opportunities the log helped us to identify. We also decided to set ourselves a team goal for 2020 to increase user attendance rate to 80%. Eventually, we’d love to eliminate no shows entirely, but for now we need to walk before we can run.

And, over the next few months, we’ll be implementing our new recruiting strategy and workflow. If you would like more details on what they are or an update on how things are going, get in touch.