When Should We Turn to Content Testing?

A primer on a great UX research method to help you design the conversation and evaluate the effectiveness of your words

By Sara Zailskas Walsh

I’ve been pretty lucky in that each of the three UX design teams I’ve worked on as a content strategist has had a UX researcher shepherding us. We’d do end-to-end research — benchmark testing, concept testing, usability testing — and, of course, each thing we’d test was chock-full of words. I’d work with our researcher to include particular questions to get specifics about the information people needed, and the entire study helped us design to meet customer needs. Important stuff.

But I always wanted to ask more than we ever had time for. Are we getting it right with the specific words we’re using? And is our definition of what’s “right” actually the same as our users’? What do they think of our claim that we’ve got your back; does our narrative make a customer feel that way?

My questions were about what we were communicating and how well it was understood, versus how usable the whole design was, which I began to realize were different than some of the more common UX research questions we tackle.

Then, a series of articles and case studies led me to fall for content testing. I started setting up content tests on my own, and with that came really great questions from UX designers I respect. One question in particular recently challenged me: Why is this a valid way of testing if the sum of all design parts makes for a great experience?

My team’s UX researcher and I sat down to answer that question, pulling from our own experiences and that of my fellow content designers here at Capital One.

Now, here’s how we answer great questions asking what content testing’s about and what we consider when we think it’s a method that will help us.

What is Content Testing?

Content testing asks users to respond to copy only — no visuals, no interactions. Just words. Think of content testing as prototyping the conversation we have with customers. Like other research methods, this is a method you can use if you decide it’s right for the types of questions you have.

In a content test, we present a customer with copy—printed simply on paper—and then ask a question related to how and what we’re communicating, such as, “Using your blue highlighter, highlight anything that you like,” or “Using green, underline anything that makes it feel like we ‘get’ you and your business.” Then we ask people to talk about what they underlined and why.

When I first started at Capital One, we held an informal research event to get to know small business owners. I pulled copy from throughout our experience and ran a content testing activity so I could begin to hear customers react to the copy we were using. It was a simple way to get familiar with our copy, consider our language in context of the communications goals I was getting to know, and listen to customers talk about how they want us to speak to them.

Why Use Content Testing?

Content testing methods help us understand the conversation we need to have with our customers both across a journey and within its specific moments. The results help us discover the conversation framework — and that framework drives the design of that customer experience.

Here are some of the questions this method has answered for our teams:

• Is this the progression of information the user needs in the end-to-end process?
• What touchpoints do we need? What should we say to customers at each moment?
• Does the framework we think we need match the customer’s mental model? Are we starting our designs at the right place?
• How do customers think of this or say this?
• What content principles might we follow when we design this experience? (Or, are the principles we hypothesized correct?)

When Should We Use Content Testing?

Content testing can help at each stage of a project for different reasons. If I could pick only one point in a design process to do content testing, it would be the concept phase. But it can be valuable in the final stages of design too — as long as usability isn’t the goal of the content test.

Early on in a design project, content testing helps us determine the holistic conversation we need to design. The findings help us figure out how we should speak to customers by understanding the language they use. On a micro level, content testing can also tell us what information a user needs and wants for the use case.

For example, we’re beginning to redesign our error messages throughout our small business banking experiences. We follow a framework to create error messages that help customers understand what happened and what they can do next to move forward or complete their task; we challenge ourselves to write for each specific use case. But, there are times when we can’t give the level of advice or guidance we would like.

There are many ways to handle an error message like that. Do we offer options even if they might not apply to the customer? If we’re going to be transparent, how much detail do people want? Should we apologize? And so on.

I wanted to do the best we could by our customers, so I set up an unmoderated content test online where I presented 10 small business owners a very general banking scenario and showed them an error message under the pretext it appeared suddenly, mid-task. I tested extreme versions of what would still be considered a solid, basic message.

Content testing delivered a key insight for writing an error message in that situation that would reassure customers and help us help them complete their task. We now understand what our message needs to include when we need to provide a general response, and we can properly design for all the layouts and spaces this message would appear. We built on our existing framework.

In the middle of designing an experience, content testing can help us understand if the words that are “sticking” in our design after several iterations are meeting our communication goals and uncover whether the words we’re using confuse or clarify.

We found that using the word innovation, for example, was well-received by half of participants under the condition it’s used properly; the other half had a strong aversion to the word — regardless of context — because of how often it’s used. Knowing that, we can stay away from that word as we write for new experiences.

After the experience is designed, content testing can help us understand if the words we’ve finalized after all those iterations are still meeting our communications goals. Note that it’s less likely that you’ll use content testing for micro-moments at the end of the design process, as content will already be incorporated into the UI usability testing, and the sum of all parts is important to test at that point.

At the end of an online account opening project, we made a last-minute decision to delay our final usability test. Our participants were already confirmed, though, and research studies ain’t cheap. So at the last-minute I put together a content test that asked customers to read through all the copy in the experience, including sample email messages they’d receive. Our goal: to confirm that through all our design iterations we maintained our narrative in our copy and that it left good impressions — the brand impressions we wanted — on customers.

We asked customers how they felt after reading all that copy without interruption and to describe the overall feel of what they read. Without being directed, customers used the same words we’d used in our communication goals. We didn’t ask them to isolate particular words right away. Instead, we asked questions and had them point to examples so we understood why. Because their responses were right in line with our goals, we felt comfortable that we didn’t need to iterate on the copy and we had maintained our narrative.

Although it’s true that the design experience as a whole — visuals, interactions and content working together — leaves an impression on customers, on that day, we used content testing to validate the effectiveness of copy’s role in the experience (and we didn’t waste an opportunity to talk to users!).

When Should We NOT Use Content Testing?

Content testing is NOT usability testing—all aspects of a design contribute to how well a space is used and perceived by customers. The UX designer who challenged our thinking is totally right in that everything from hierarchy to visuals contribute to how a customer perceives an experience.

And that’s a key point: Content testing doesn’t solve for information hierarchy, formatting, and visuals — or, in other words, how a design is delivered. (That comes next!)

How Does Content Testing Help Design?

Here are three big benefits:

1. It helps designers establish the framework for our conversations with customers.
2. It helps designers understand the words we need to use so customers understand us.
3. It helps designers understand the information and emotion their designs need to convey in customer moments.

We use content testing regularly at Capital One; it’s one of the reasons I joined the team. Our fraud team has used content testing to test confidence and confusion in emails they draft, and our bank team uses content testing to identify a framework for emails and notifications. Our credit card team used content testing as a tool when tackling customer onboarding after hypothesizing how to best guide new customers. They weren’t sure if their ideas matched customer needs, so they used content testing to evaluate their ideas.

You can do it!

If you think content testing can work for your team, try it out. List the questions you have and determine if they’re about usability or framework. And to see a non-Capital One example of what setting up a content test looks like, check out the UK.gov team’s article that triggered my love for it, “A simple technique for evaluating content.”

Sara Zailskas Walsh is a kale-hating content designer for banking products for small business owners at Capital One in San Francisco. Did we mention we’re hiring? :)