Why Ecological Validity Matters for UX Research

As a UX researcher, you know your research methods. You’ve done contextual inquiries. You’ve done usability benchmarking. You’ve done A/B testing, clickstream analysis, ethnographic field studies, and everything in between. But is your research ecologically valid?

Psychologist Kurt Lewin first introduced ecological validity in the 1940s. He noticed how many psychological experiments tend to pluck research subjects out of their everyday environments, place them in unfamiliar settings, and act as if their reactions were an accurate representation of behavior “in the wild.” However, he recognized that the less research conditions simulate reality, the less they will accurately model how people actually behave in reality. Hence, ecological validity is the extent to which research findings are generalizable to everyday life.

To borrow a more familiar idea, ecological validity is a measure of how organic research is. Just like organic foods are free from strange dyes or chemicals, thoroughly organic research is free from factors that might skew, alter, or unconsciously influence the natural behaviors of a research subject.

My favorite example of a failure to perform organic research is an experiment that wanted to find out how long it took people to remember the date when prompted. Two researchers stood outside of a university building, one with a clipboard and one with a stopwatch. When someone would leave the building, one would ask them the date while the other timed how long it took for the subject to respond.

Sounds like a good experiment, right? When we account for ecological validity, this otherwise interesting experiment begins to seem less and less valid. First, researchers cornered and surprised unwitting participants as they were walking out of a building. Second, without explanation, one of the researchers began a stopwatch immediately after the research subject was bombarded with a surprise question.

Both factors dramatically reduce the ecological validity of this experiment because they do not simulate real-world conditions and the researcher’s presence likely influenced the results.

While the goal of this research was to find out how long it took someone to recall the date, what the researchers really examined was how well college students recall the date after being cornered and timed by strange people lurking outside of a university building. In other words, almost no aspect of this experiment tried to replicate real-world conditions.

A more ecologically valid approach would observe how quickly a cashier provides the date when someone is writing a check, or a real estate agent when someone is trying to sign mortgage documents. Both are cases where someone might actually need the date. In addition, hiding the stopwatch would also help make this experiment more ecologically valid. Being timed puts a lot of pressure on someone, which might skew the findings of the experiment.

Why Does Ecological Validity Matter?

Ecological validity matters because the results of your UX research may skewed, or worse, totally wrong. Consider a lab-based, standard usability study. Participants are pulled into an artificially constructed setting and told to complete a series of tasks under observation. In reality, none of your participants would encounter your product that way.

Maybe they use your product sitting in their undies, half awake at 4am in the morning. Maybe they use your product holding their crying baby. They definitely wouldn’t follow a series of artificial instructions, or deal with a researcher looming over their shoulder.

Instead, people use your product organically, and if you don’t account for ecological validity when performing usability testing, you might be making decisions based on bad information. There are some practical ways that you can fix this problem, however.

In order to achieve high levels of ecological validity, there are some principles UX researchers can follow:

  1. Develop tasks users might actually do, unprompted. Learn more about individual users and what kind of people they are, what sorts of needs they have, and why they might use your product. Direct them to use a product as they might in the “real world” as much as possible.

Some UX research methods do not lend themselves to high levels of ecological validity. For example, I’ve seen some forms of A/B testing that show a single user both versions of a UI, and ask them to either give their preference or complete a task in both versions. Users almost never encounter two similar versions of a website, back-to-back, in their everyday lives. Not only is this not a real A/B test, it’s not ecologically valid.

On the other hand, contextual inquiry and ethnographic studies can have very high levels of ecological validity. When performed on-site with minimal interference with the research subject, both methods of UX research will provide rich insights into actual user behaviors. In addition, both research methods ask users to go about their everyday lives, so there are less ways in which their natural behaviors might be unwittingly altered.

In both contextual inquiry and ethnographic research, the choice of using audio or video recording has will impact its ecological validity. Both present tradeoffs. Audio recording is unobtrusive and easier to use in settings where privacy or confidentiality is important, making it more ecologically valid. Audio recording also requires a user to speak out loud, and fails to capture details that video can, providing less ecological validity.

While this problem can be somewhat ameliorated by taking notes, note-taking might make some users nervous, altering their natural behaviors. On the other hand, video can be extremely intrusive and might require special permission to use in some workplaces, making it less ecologically valid. But, if the video recording is done discreetly, it will produce a much higher level of ecological validity than audio recording because it can capture more of the natural, unfiltered behaviors of users. A researcher can leave the room entirely and the user is likely to forget they are being recorded.

Artificiality will always be a part of usability testing, but striving for high levels of ecological validity will produce more accurate representations of how and why people use your product. The next time you decide to perform UX research, think carefully about how you can limit your interference with real user behaviors, and try to capture situations in which real users organically use your product. The results might surprise you.