User Research — How To Tell If Your Participant Is Faking It
Awesome! You’ve just completed a long day of prototype interviews, got a ton of great feedback on designs in the works, and bought back impactful insights to the team.
Now it’s culminating in this week’s product release. But wait…somehow this isn’t at all what you were expecting to see. Analytics show crazy usage patterns, your contact inbox is full of unanticipated issues, and one user review claimed your product somehow maimed an adorable kitten somewhere in this cruel, dark world. What happened?
One of the main culprits of mismatched expectations and a sore user researcher is heinously simple: your research participants were faking it. Or at least not being completely honest with their answers and actions during your research sessions.
So how can you tell if your research participants are faking it so you may never harm cuteness again? Here’s a list to get you started…
- Is there a reason your participant may not feel comfortable? Even before the session starts you can take a quick mental check to see if your research participant has any reasons to feel uncomfortable with being completely honest during the session. You’ll find a quick guide on reasons and rationale here.
- Your participant is changing their opinions and actions. One of the clearest signs a participant isn’t showing you their true selves is that they keep changing what they say and do throughout the session. For example, if you ask “why did you decide to click this ad?” and their response is “Well I wouldn’t actually do that I guess…” then you may have shape-shifting user data on your hands, and it’s going to be your job to get to the actual answer. A good series of probing questions here can go a long way in uncovering contradictions and figuring out why they exist, so you can resolve discomfort and get back to the real stuff.
3. Your participant reflects in the 3rd person. If a majority of the feedback your participant gives includes phrases such as “Some people might…” or “I have a friend who would love this…” or any other reference to someone other than themselves, then you’re probably not getting great data. They’re not exactly faking it or hiding anything, but they’re definitely not giving you relevant data about their thoughts, feelings, and behaviors. Be sure to clarify whether they mean to really speak for themselves or not, and if not, find out why.
4. Your participant gives you quick answers with little variety. At least once in your career you will get a participant with a ton of enthusiasm, telling you they’ve been a loyal user for years (you just released last year), jumping on any and every opportunity to sing your praises. Other times, you’ll get a complete contrarian who seems to hate everything you’re proud of. Worse yet, and most insidiously, you’ll find someone who just doesn’t want to think. They will rush, flash you a smile, and tell you everything is “usable” or “convenient” or some other thing they think you want to hear. The common thread between all these people these people is they aren’t really taking time to engage their emotions and give a thoughtful response. They are just trying to get this session over with. Slow down and take some time to ask the “why” questions with participants like these.
5. Your participant takes the time to think. On the other end of the spectrum is the participant who is a little too thoughtful with their responses. They clearly take the time to think about exactly what they’re going to say, and maybe also how they’re going to say it. While not all participants that take a long time to think are trying to avoid being natural, it does happen from time to time and a good way to tell the two apart is that the ones who aren’t being natural will often take clear and deliberate pauses. If the last guy gave you automatic canned answers that were prepared, this gal may be preparing them on the spot. In the end you want a natural response, so asking surprising questions that don’t offer much processing time would be helpful here.
6. Your participant implies actions rather than doing them. Sometimes you go through a prototype session and your participant sees a screen and says, “oh yeah, I would do this stuff…” or they may see a button and remark, “I’d probably click that…”. A few seconds pass and you realize they won’t actually do it, they just wanted to tell you they would. This is a clear sign your participant is either telling you they’d behave in a way they actually wouldn’t or they are too uncomfortable for whatever reason (privacy, embarrassment, etc.) to go forth with the action. It’s up to you to figure out which. Ask why they didn’t just do the action and if they would really do it, as sometimes participants just need to be nudged into a more thoughtful state to really evaluate their behavior against their words.
This list is by no means exhaustive of all the signs a participant may be giving you inaccurate data. They are the ones I find most common and should help you get a majority of the noise out of your interview sessions.
In the end, what you’re looking for in an honest and natural participant is variety, organic flow of thought and informality. A person who is being truly honest and open will give detailed personal reflections, express themselves through your prototype and generally surprise you by what they do and say. Happy interviewing. Bring back the real stuff.
You can check out what Boris is up to at www.borisbee.com. We’ve all got something to say, come say it!