Cross-Examining Design Research

Being a product designer who conducts and uses insights from user research is difficult. There’s a lot of industry hyperbole, “rules” from old-timers, and powerhouse design team examples to live up to. While most product designers are eager to talk to users and are mindful of being biased or asking leading questions, when it comes to testing ideas–they don’t really understand principles of testing. They either hold themselves to some kind of double-blind medical research standard or dismiss quantitative testing as a mystical numbers game.

“Because of this, designers tend to separate the world of qualitative analysis from quantitative analysis — isolating the value of each within specific procedures and methods. But these methods of learning don’t compete. They aren’t mutually exclusive in merit or process.”

Almost 5 years ago, Lucid increased their investment in design along with their investment in analytics and A/B testing. Since then, all designers on my team have learned how to incorporate statistical reasoning into our design process. It wasn’t easy at first — most of us were uncomfortable trying to rationalize the user experience through metrics and testing. But now, our team can be found improving the product with strong design hypotheses and questioning unexpected test results. We’ve tried to embrace both methods of learning for their strengths and have improved our design process by combining principles from both.

Foundations of Testing

For many designers, stepping into the world of statistical reasoning is unnerving. It’s easy to get lost in unfamiliar methods and nomenclature: statistical significance, p-values, parameter decisions, confidence interval, power calculations, etc. Exact definitions and usage aside, what these terms (and others) really represent are important concepts of reality that underpin all types of testing… both qualitative and quantitative. They are –

• (A) What’s possible in the world
• (B) What you’re able to measure (in the world)
• (C) Your perception of what was measured

Understanding these states of reality is important because they help us understand testing principles and dictate the processes we use for testing. Even though these paradigms are easier to make sense of in quantitative testing, it doesn’t mean that they can’t be applied to qualitative methods. In fact, familiarizing yourself with them can help you rethink your qualitative research processes, trust the research you have performed, and make insights more actionable.

Chance: What’s possible in the world vs. what’s measurable

One of the most important paradigms to think about in testing is the relationship between (A) What is possible in the world and (B) What we’re able to measure in the world. Think about tossing a coin, with each toss there’s a 50/50 chance it will land heads or tails. But if you flip the coin a hundred times, you’ll probably not measure an exact split of heads/tails even though the possibility of the 50/50 exists with each toss (odds). In reality, you might see heads land 47 times and tails land 53 times. And if you were to run a test of 100 tosses over and over, you’ll continue to see similar variation. Let’s look at a different scenario. If you flip a coin 100 times, and heads lands 30 times and tails lands 70 times, you may come to the conclusion that this measured difference is no longer attributed to “chance”.

The point at which measured variation is no longer attributed to chance isn’t a magic number. Identifying the threshold between chance and advantage represents the level of confidence (not truth) that one has in what they can measure. And to be effective, it has to be intentionally chosen before the test begins. Sure there are some industry standards for analysts to use, but again–it’s not the number that is important, but the concept that the number represents.

Understanding this threshold helps you make sense of variation in data–which is something all designers encounter when testing ideas and product decisions. This is important in both qualitative and quantitative testing.

If you don’t have confidence that the variation of data you measure is attributed to chance or advantage, you may interpret an advantage when there isn’t one.

You may think there isn’t a problem when there actually is one! You might throw out a unique use case that drives innovation because you labeled it as an edge case…

When designers engage in qualitative testing, they don’t usually have statistical power working for them. So instead we rely on “saturation”, pattern matching, and we quote Nielson Norman. In reality these apologetic approaches are trying to mimic quantitative testing, without holding ourselves to the rigor and confidence of quantitative analysis. These are not sound methods to bifurcate chance from advantage.

Bias: What’s measured vs. what’s perceived

Another important relationship that designers need to understand is the relationship between (B) What you’re able to measure and (C)Your perception of what was measured. Put another way, what is the relationship between the existence of conditions and the perception of conditions. Making a distinction between these two things is important because it reveals possible outcomes of interpretation (test results).

If you perceive the existence of a condition when there is none, this is called a false positive. If you don’t perceive the existence of a condition when there is a condition, this is called a false negative. Applying this to qualitative research is to say, if the designer perceives a signal in feedback (what’s measured) that doesn’t exist, or misperceives a signal that does exist… they will think a research participant is saying something they aren’t! This is where things for designers get tricky!

Qualitative research isn’t research in a controlled setting or lab where everything is measured and instruments are calibrated. Things get infinitely more complex when humans are interviewing other humans about their experiences of interacting with objects in the world (phenomenology). Factor in cognitive psychology, and you end up with other dimensions like motivations, tasks, senses, attention, short-term memory, long-term memory, and response… and something as simple as A/B testing ends up looking like a big bowl of steaming, philosophical soup.

When humans provide the measuring stick for other humans, chances are there’s bias involved–even when you do everything you can to interpret responses and identify truth.

You try not to ask leading questions. You talk to “enough” customers to see similar responses. You’re looking for something to learn–to validate and drive change. But what other people in your organization want to know is, what confidence do you bring to the table that you interpreted your research data correctly? How do you convince them that what you heard in your 12 interviews represents (A) What’s possible, from (B) What you measured, from (C)What you interpreted?

Motivations before biases

Since most of us never interview enough participants to reach any amount of statistical significance, and because each person will understand and respond to research prompts differently… it means the information gathered from your handful of interviews will likely exhibit a high level of variation. This, in turn, means you are likely to misinterpret responses and data.

Trying to be unbiased when some of your participants are either misrepresenting information or you are likely to misinterpret information, increases the odds of introducing false positives and false negatives into your findings.

False positives and false negatives are bad things to have in the equation of making business decisions. Simply wanting to be unbiased, sticking to an interview script, and not asking leading questions may not be enough. While we may not think of ourselves as biased, we are motivated beings. A business is motivated to convince others to exchange $$$ for a product or service. Your job as a designer is to find out what motivates a customer to pay you each month for value—and what prevents them from paying.

If this is the situation under which most product designers conduct qualitative user research, then we should probably throw double-blind clinical trial methods out the window. To really avoid bias in qualitative testing, you truly can’t care about the outcome of your test, and your participants couldn’t ever know they are being tested!

Maybe, instead of sticking to a strict script and worrying obsessively about the way to ask a perfectly neutral-sounding question, we should be more concerned about a research participant’s ability to confidently “stick to their story”?

Your research participant didn’t swear on the bible before offering their testimony, they took your $. That’s called bribery. This is a courtroom and it’s all about cross-examining the user who took my $25 Amazon Gift card for spending 30 minutes to talk with me!

An argument for cross-examination

Don’t get me wrong, I’m not proposing designers become jerk lawyers whose client’s innocence or guilt is on the line, but I do think it’s OK to politely and intentionally lead you down one path of rationale and then another. I will tell you something in the experience is broken and doesn’t make sense to me — when maybe it does. I will tell you that the designer next to me has a different opinion than I do — even if they don’t. If they tell me they like something, I dig in and find out why. If they tell me something won’t work, I try to understand the conditions and situations when it will work! As designers we should be more concerned about playing the role of prosecutor and defendant more than medical researcher.

What I’m trying to do by cross examining is two-fold. First, I’m trying to understand their propensity to change positions — which is to say, a participant’s natural variation (chance) vs. situational conditions. I’m also trying to find out if a participant is easily persuaded. If a participant can’t describe conditions or situations for an outcome, or if they keep changing their mind, you should probably throw out their testimony altogether. They are an unreliable witness! Remember–think courtroom, not cancer research.

Secondly, a cross-examine approach usually leads to more of a conversation-style interview, wherein all the participants can be more reflective about their position (rationale), allowing for more synthesis between participant and interviewer. It shouldn’t be a debate, but rather dialectic in nature; which is a discourse between two or more people holding different points of view about a subject–but wishing to establish the truth through reasoned arguments.

Preparing the witness for the courtroom experience

While using a courtroom metaphor is useful to articulate an approach to this type of research, let’s be honest–no research participant wants to feel like they are in a courtroom! Setting up your participants to be successful with a cross examination approach takes some attenuation. You will likely will need to outline expectations and etiquette for the proceedings. Below are some suggestions for facilitation your understanding of (A) What’s possible in the world, (B) What you’re able to measure (in the world), (C)Your perception of what was measured.

• “Have you ever seen your coworkers or others in your industry use our tool for something interesting or unique”? — What’s possible in the world
• “We might repeat the same questions throughout the course of this interview (to give us more confidence in your responses). Feel free to push back and hold your ground on a topic that you feel strongly about.”—What you’re able to measure
• “Let us know if any opinions you expressed are strongly / loosely held… it will help us gauge your own confidence in your answers.”—Your perception of what was measured
• “You didn’t seem very interested in that last feature we showed you… are there any situations or conditions where you would change your mind and be excited to use that feature?”—What’s possible in the world

Deeper learning

Many times when using this style of interviewing, a participant will feel challenged, change their mind, then finally hold their position with strong reasoning; and you understand rationale that’s been articulated and tested. Likewise, many times we’ve walked away with our assumptions overturned… and not because a “user told us so”, but because we really understand where we misunderstood needs and situations. And if either party in this dialectic process changes their point of view during the course of the research, we at least walk away with an understanding of the conditions under which a user will change their mind.

This framework and techniques produces the insights needed to build confidence and drive change where needed. And isn’t this what we really want from the experiences of research? And isn’t that what the organizations we work for really want as well?

Again, I’m not advocating for biased research. I’m advocating for strong rationale (answers) to the design questions we have. This allows us increased confidence in proposed solutions! Be bold. Don’t accept answers at face value. Discover insights that lead you to action.

Learning the discipline of quantitative research and analysis has unlocked concepts to improve our qualitative efforts as a design team. We’ve found A/B testing principles creeping into our own discovery frameworks, helping us become better researchers. I really believe that the future of product design is a world where qualitative reasoning and quantitative reasoning come together and are used more expertly to better understand the people we design for.

(This is a post from the UX design team at Lucid. We make collaborative visualization and brand templating tools: Lucidchart and Lucidpress, and we’re currently hiring designers!)