Qualitative Research, Bias & The Hunt For Truth

By Anna Ho, Associate Strategist

I once heard qualitative research described as the “explicit knowledge of bias.” Anyone who’s been involved in human research has probably encountered the pitfalls of bias. It’s terrific for validating what we already know, but terrible for helping us get at what we don’t know. When it comes to mitigating bias, my approach is this:

Don’t ask people to tell you what they think. Ask them to tell you what they know.

In my former life as a teacher, my aim was to help students get at what they know and in turn, use that knowledge to motivate my students to explore and learn subject matter. Now, as an Associate Strategist at SI, part of my job is to help users get at what they know and use that knowledge to help the team at SI craft meaningful digital experiences that motivate and resonate with users. In getting at what people know, we have the opportunity to craft a baseline of sorts — to get at user assumptions and their truthful understanding of what we put in front of them. By asking users to tell us what they know, we push ourselves as researchers to truly listen to the user and not just limit ourselves to looking for answers to pre-defined questions.

As it was with teaching, the challenge of helping people get at what they know is that people are typically focused on what you want to know. They are not often in the habit of stating what they know. In the diverse range of user research we conduct at SI, we encounter a variety of personalities, but more often than not, participants fall under one of two archetypes.

Chatty Cathy:

Cathy aims to be helpful. She is a fountain of ideas and opinions, eager to help you make your product better. Ask her what she thinks, and she will do just that, seemingly narrating every single thought or question that comes to mind. Cathy is the ideal research participant. She is cooperative, articulate, and more often than not, self-aware. The challenge with Cathy is that like an attic full of hidden treasures, we have to sift through a lot of clutter before we can determine what’s worth keeping and which thoughts are better tossed aside.

Silent Sammy:

Whether he’s there because he’s getting paid or someone he knows asked for his help, Sammy feels obligated to participate. He is minimally cooperative. He wants to give you valuable feedback, but he’s not going to go out of his way to tell you anything you didn’t think to ask him. Typically, Sammy’s responses are no longer than a tweet. You can try asking him “why?” but it’s unlikely to elicit much more insight. The challenge with Sammy is that like some mystical sage in a fantasy story, the key to getting useful information from him is to ask the right question.

Whether you’re interviewing a Chatty Cathy or a Silent Sammy, your aim should be the same; get at what they know. Here are some strategies to help you do this:

1. Define objectives. | Objectives are what distinguish an interview from a conversation. Before inviting users to tell you what they know, it’s important to identify specific research aims that will help you set up parameters during the interview. For example, do you want feedback on a particular feature or concept? Do you want to gain some contextual insight around the use of your product? Do you want to learn about the values of your customer? Clearly established research objectives will help you to define the context of your user interview and give you the confidence to let your users get at their own thoughts.

2. Listen. Really listen. | This one may seem like a no-brainer, but listening can be tricky when you have clearly defined objectives in mind and only 30 minutes to get at what you need. You’ll know that you’re actively listening when you find yourself striving to understand what your user is thinking rather than asking yourself if the user answered all the questions listed in your script.

3. Don’t be shy about saying, “Tell me more.” | Whether you’re dealing with a Chatty Cathy or a Silent Sammy, sometimes the user will speak with such a definitive tone that it seems like they’ve told you everything there is to be told. Don’t be afraid to prod them a bit more, even at the risk of them repeating what they already said. Often times a golden nugget of truth appears in this rehashing or elaboration of thoughts.

  • To Silent Sammy, I might say, “You said ________________. Tell me more. What made you say that?”
  • To Chatty Cathy, I might say, “That’s all interesting, but I’d like to hear more about when you said ________________. Tell more about that.”

4. Give them something to talk about. | Asking a person to tell you what they know can be quite a daunting order. To make this task less intimidating, provide users with something to look at — a prototype, wireframes, cards, sketches, etc. Be sure to select stimuli that is narrow enough in scope so as to focus the attention of your users, and, more importantly, support your research objectives, but is also rich enough in content to allow your user some room to explore and wonder over. Put the stimuli in front of your users and ask them to tell you what they notice. If the person is at a loss for words, have them literally point out what they notice. By getting them to highlight what catches their notice, you now have the opportunity to ask them about why that detail caught their attention. Moreover, by having them explain their thinking, you can work on understanding the user’s underlying assumptions, priorities, and general understanding of what you’ve put in front of them.

Be sure to take note of what users ignore or pay no attention to. This may be an indication of points of confusion or of low priority for the user.

This focus on getting at what people know is a core part of the way we approach user research in the Strategy department at SI. We’ve found that by uncovering the underlying beliefs and knowledge of users we’re better able to anticipate hurdles that will impede users from engaging with the product and/or experience. This insight, in conjunction with what we know about user behavior, preferences, and motivations, enables us to build products and experiences that more authentically meet the needs of users.


Originally published at smashingideas.com on June 29, 2015.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.