Know Your User

A quick-hit on recruiting for user research

Introduction

Don Norman once said that if there is any principle that is sacred to those in the field of user-interface design and human computer interaction, it is ‘know your user.’ This sort of mantra for what is commonly known as User Experience (UX) Design has guided the field for decades. UX methods are mostly all centered around aligning the needs of the user with the capabilities or features of a product. Where defining the business requirements and constraints come from the business side, defining the user requirements and constraints comes from the UX side. And so, the wheels keep on turning.

So naturally the question becomes- how do we understand the user? Again, it typically depends on the nature of the project. There are many books out in the world that do a pretty good job of breaking down user research methods into specific categories. Universal Methods of Design by Bella Martin and Bruce Hannington is a good example. Nielsen Norman Group also outlines the different user research methods quite well. But what I think one of the big take-aways from this piece is- base your recruiting around your research questions and hypotheses, or, don’t recruit blindly.

At Balance we make sure that our samples are balanced based upon behavioral and attitudinal segments. This manifests itself as a wholesome sample of research participants that are aligned with the purpose of our research. We arrive at these segments based upon the characteristics of the product we are designing and the people who will be using it- sometimes going so far as to test two or three user groups in the same research project.

This article will take a more general approach. It will serve as a primer for different research methods and how they influence the recruiting of research participants. I’m going to try to be brief with my explanations of these for the purpose of this article.

Behavioral vs. Attitudinal Research

When defining research methods it is important to ask the important question for any UX designer — “Why?”

Why are we doing this research? What are we trying to accomplish? What are our goals as researchers- which in turn begs the high-level question- “What are our user’s attitudes and behaviors”

So this all boils down to two extremes:

Behavioral Research: Done to understand what people do. This research is typically meant to help see how people use a product, as opposed to what they say about it. There’s a discrepancy between the two.

Attitudinal Research: Done to understand what people say. This research is usually done to understand or measure people’s stated beliefs and is very popular among marketing departments.

The benefit of breaking down methods into these two major categories, again, goes back to the goals of the research and can be used to help define the sorts of users you want to work with. It goes without saying that attitudinal and behavioral data can be extracted from the same study (e.g. field visits and usability studies), but understanding the difference between what a user says and what they do is essential. Defining these methods can also help a team arrive at different behavioral and attitudinal variables which can be used to arrive at new research questions or hypotheses.

Qualitative vs. Quantitative Data

So you’ve selected your research method and now you want to decide what sort of data you want to collect. There are a couple of routes for this which are qualitative and quantitative, or a combination of the two.

Christian Rohrer of Nielsen Norman Group breaks these down nicely:

“studies that are qualitative in nature generate data about behaviors or attitudes based on observing them directly, whereas in quantitative studies, the data about the behavior or attitudes in question are gathered indirectly, through a measurement or an instrument such as a survey or an analytics tool.”

So, going back to the topic of this post- “who are you recruiting?” you need to ask yourself, “why are we recruiting them?” Do we want to understand what they think about a product? Perhaps an Attitudinal Qualitative study (e.g. User Interviews and Card Sorts with a deliverable of a Kano Analysis as an output) is in order. Do we want to understand how quickly they can complete a task? Perhaps a Behavioral Quantitative study (e.g. a Usability Test and Task Analysis with a Key Performance Indicator matrix) could be applied.

Again, it all boils down to what you are trying to accomplish. For example, if you’re trying to understand how understandable your product is to someone who has never used it- you might want to recruit people all in the same basket of characteristics in order to eliminate extraneous variables such as familiarity with technology (in the case of digital products).

Evaluative, Generative and Exploratory

I want to briefly cover the big three types of research methods as they relate to recruiting. This is important because typically specific research methods are used depending on where you are in the design process. Let’s start with evaluative.

Evaluative

Generally used to evaluate the performance of a design, system, or product- whether it’s what people think about it or how quickly they can use it. The fidelity of the prototype of product generally does not matter (unless you’re evaluating visual design), you just want to evaluate how it is working. For recruiting, you may want to evaluate it with novice users (people who may never have seen or heard of such a product), advanced users (people who use similar products or systems and have some degree of familiarity), or power users (people who regularly use your product or system). If your goal is just to find issues with the existing system, you could probably just test with 4 or 5 users. For statistical significance in evaluating a product (popular with quantitative studies), typically 30 participants is a solid bet.

It’s important to be careful, however, of falling into the trap of recruiting based upon demographics such as age. To illustrate this point, consider this quote:

Empathy over stereotypes…. Demographics inject bias because we stereotype people by their demographics. Old people are like this, millennials are like that. We’re biased by demographics.

Generative

Popular among the divergent phase of a design project are generative studies. Generative research is done to develop cognitive empathy for the people that will be using our designs. Many times it’s qualitative research and it can be done in order to identify all sorts of opportunities for a project.. For example, if you or your team are trying to come up with new ideas for a product, perhaps going out to where users of the product typically congregate and asking them questions could help generate some ideas. This would be a field visit with guerilla research, or even a fly-on-the-wall study. Ultimately, the goal here is to create ideas, and then evaluate which one of those ideas are the most effective. In this respect, if you are trying to generate ideas, make sure to position yourself around participants who are most likely to use the product. Even a user interview with a related participant can help you do this.

Exploratory

Exploratory research is typically conducted to look into a finding, problem or issue that needs to be put under the magnifying glass, or studied more closely. For example, a researcher might want to understand why something is important to a user. To do this, they could pull in a variety of users relevant to the project and ask them to bring in something that’s important to them. Sometimes this can also be done with direct observation, like watching a user struggle to complete a task, or seeing what areas of the product provide the most satisfaction. Then, the researcher asks “why” in order to explore these answers. Why are the user’s experiencing frustration? Why is this important to them? Why do they do it this way compared to others? Answering these questions can take designers closer to solutions and further away from confusion. So for recruiting, it’s essential in this case to recruit users that can help to explore these questions. Do you want to explore why something is difficult to new users? Recruit novices. Do you want to explore what is important for power users? Recruit power users.

Conclusion

I hope this article provides good insight into not only who to recruit, but why you should recruit them. To summarize, start out with research questions that ask what you are trying to accomplish. Identify the research methods that are most applicable for helping you answer those research questions. Finally, decide on the type and number of users you need to speak with in order to reach that next step. Research is an ongoing process, especially in design, which is why it’s necessary to constantly ask “why” and refine your methodology. Research that is too straight forward or cookie-cutter (beware: tunnel vision) might not be adaptive enough to get to the truth of the questions you are asking. As always, maintain an open mind and learn.

Sources