What UX people can learn from journalists
— or why surveys are not a design research tool we should use.
Surveys allow collecting large amounts of data in a relatively short time. This is why there is conventional wisdom that Surveys are easy, surveys are cheap, which is actually not true. A good survey design even often means investing in pre-tests to ask questions that will lead to meaningful answers. However, these beliefs are the reason why there is a high tendency to use surveys in design research. But this is a fallacy.
Surveys might be fine when you might want to know about the demographic structure of a population or basic opinions aka trends to follow up on already known topics. But we suggest to not use them in UX research meaning base your design decisions on surveys. Here’s why:
There’s something called the „social desirability bias“ which is the tendency to respond in a way that makes people look better than they are. For example, a survey respondent might report that they engage in more healthy or „better“behaviors than they actually do. In interviews or — in this case even better — user observations or diary studies you can get around this way better, as you observe interviewees reactions and/or by asking better follow-up questions.
Surveys are also designed to be standardized. This means everybody gets the same questions and — this is the point — also the same options to choose an answer from. Standardization is important for surveys so that the results can be generalized to a larger population due to comparable answers. To achieve that, surveys most often use close-ended questions, providing people with a range of possible answers: „Are you satisfied with solution A? A) Yes; B) No. Or: „Which color do you like most?“ A) Red; B) Blue; C) Green. So, in the end, the results are quantifiable: „35% said that they like blue, but 75 % prefer green“
But they do not tell us why people don’t like blue as much as red. Sure we can implement those questions in our survey and have some fun mixing up qualitative data with quantitative data when it comes to evaluating the 500 or even more results. However, we cannot individually address the answers of individual people and ask targeted follow-up questions to that specific answer and dig deeper. And why all the reluctance to sit down and do 10 in-person interviews that are guaranteed to give more insights into a topic?
Open-ended questions vs. closed-ended questions
In contrast to closed-ended questions used in surveys, open-ended questions are intended to give the interviewee space to provide us with more detailed answers. For example “Why do you think you like red?“; or: „Why don’t you like blue (…)?“
Using those open-ended questions is especially important if you aren’t familiar with a topic, or you need some background information on that topic / dig deeper. Also, we see *how* people react: non-verbal cues are also important information. This all provides valuable information we do not get with a survey. This kind of information is important, especially when we start in design/topic research and so start building hypothesis.
Unleash your inner Journalist
When you listen to someone and let them speak using their own words, you will get new insights which will lead to new possibilities and opportunities that you may not have considered by only asking close-ended questions.
You also get insights into their thinking — into their mental models, and the vocabulary they use to describe things. Journalists often start their conversations with people with a very simple open-ended question: „Tell me a little bit about yourself“; „How often do you use XY?“; „Why don’t you use XY?“; „Tell me more about that experience“
Using open-ended questions can also be good follow-ups to closed-ended questions like: „Do you like this layout?“ (options: yes; no, don’t know); „What exactly don’t you like?“ „Can you describe it a little bit more?“; „Why do you think this might not work for you?“
By asking open-ended questions you allow your interviewee to expand upon why they think this specific layout/thing/whatever might not work. And depending on those „expanded“ answers you can react i.e. to ask further follow-up questions which often lead to a new insight —an important benefit you would be missing out by just presenting a survey to a bazillion people and waiting for their response.
So in short, quantity is not always „better“ and „truer “ than quality. It should be used for a different purpose.
You will also be surprised about all the new insights you get through in-person interviews and/or observations, which is especially important when we conduct design research which will be the basis for design decisions.
