Know What I Mean?
How cognitive interviewing can improve survey comprehension across all participants.
Quick, think about a meaningful experience.
Did you think of something that was important, worthwhile, or special to you?
If so, that’s what we had in mind recently when we were asked to create a survey measuring the meaningfulness of a particular experience. But many of our survey participants understood the question differently. Some thought we were asking about importance or relevance, while others took it to mean how much they cared for the experience.
Culture and language can have a big impact on how participants interpret survey questions. And even among people who speak the same language in the same community, ambiguously worded questions can yield a wide range of interpretations. Needless to say, when participants are answering different questions than the ones you thought you were asking, useful insights are hard to come by.
We responded to the confusion by doing some cognitive interviewing with our survey questions. As a result, we were able to create more culturally appropriate and comprehensible questions — and to get results that were much more, well, meaningful.
What’s cognitive interviewing?
Cognitive interviewing can be an effective way to ensure that a survey is interpreted the same way — or as close to the same way as possible — across different participants, languages, and cultures. It can also help you fit a survey to your specific use case.
Also called cognitive testing, the method was developed in the 1980s as a way of testing how people understand specific survey questions. It also lets researchers uncover how people retrieve relevant information, make judgments, and ultimately choose a response.
In cognitive interviewing, participants explain — in their own words — how they understand your survey questions. Do the response options make sense? How do they weigh one option against the others? The results can reveal potential comprehension problems you might not have considered — or confirm that you’re measuring exactly what you meant to measure.
When should you consider cognitive interviewing?
Cognitive interviewing can be especially powerful when you’re developing a survey with potentially difficult or nuanced concepts. Returning to our earlier example, we tested whether people understood what we meant by meaningful. We found that people were hesitant to answer the question because the translated word could have so many connotations, including relevant, dedicated, and interested. In some languages, such as Hindi and French, the translation was roughly equivalent to important. We decided to change the word to important, which was clearer and less open to interpretation.
While cognitive interviews are no substitute for high-quality translations of your survey, they can often help you review and improve them. For example, we recently used a survey question that asked respondents whether they were committed to something. The French translation of this word, implique, suggested criminal activity, as in being implicated in a crime. The process can also uncover idiomatic or colloquial phrases that can get lost in translation, like looking forward, feeling blue, or down to earth.
Beyond linguistic gaffes, cognitive interviews can also help ensure that your questions are culturally appropriate. For example, in one recent survey of people in Mexico, we discovered through cognitive interviews that a question about personal gain wasn’t resonating because participants tended to think more collectively. We rephrased the question to emphasize benefit to others.
Another use of cognitive interviewing is to help you apply a survey developed for one product (whether from an internal or external source) to another use case. It can help you figure out whether and how questions should be altered for the new context. If you ask about a group or community in one survey, does that question make sense in a context that’s more individualistic? For example, our research found that in Nigeria, the concept of community tends to be broader and more inclusive than in the U.S.
When are cognitive interviews not appropriate?
We aren’t suggesting that you need to do cognitive interviewing for every survey. If your concepts are straightforward (if in doubt, ask a colleague), cognitive interviewing may not add much value. For example, in a recent survey we asked users to tell us who (that is, which of their friends) they believed had access to information about which Facebook Groups they (the users) belonged to. This question didn’t require cognitive testing because it was relatively unambiguous.
Cognitive interviews become more powerful as your survey questions become more open to interpretation. In a different survey that we cognitive-tested, we asked users about their perception of content. The word had different connotations to different users, so we revised the question and provided specific examples.
Cognitive interviews also shouldn’t be used as a way to understand fundamental concepts in other cultures. For example, if you’re using a U.S.-centric measurement tool with non-U.S. participants, you may need to first do some foundational work to understand participants’ specific cultural context before you cognitive-test the specific questions. In our recent work in Mexico with participants from underserved populations, this kind of fundamental work informed our cognitive interviewing.
Cognitive interviewing may add a little extra time to the beginning of survey development, but it often delivers benefits that far outweigh the effort. After all, when you can be more confident in the insights you draw from the survey responses, that’s just … so … important.
Authors: Nick Brown, Quantitative UX Researcher at Facebook; Ann Hsieh, Research Manager at Facebook; Marisol Escobar-Martinez, UX Researcher at Facebook (from left to right)
Illustrator: Drew Bardana
