Using surveys to test creative assets? You’re not doing enough.
Survey-based testing makes the creative development process efficient and effective. But scale, speed, and bias limitations make it an incomplete testing solution. Here’s why you need a robust optimization flywheel.
Previously we wrote about the need for creative asset testing. There can be a huge difference in effectiveness between “good” and “great” creative, and even the best marketers and award committees are unreliable at telling the difference.
Every creative agency needs a system to evaluate creative assets, and many turn to traditional survey-based testing as their primary tool. At Known, we recognize the value of survey-based testing, but also understand its limitations. Ultimately, survey-based testing should be one piece of a robust testing and optimization framework.
Advantages of survey-based creative testing
If used correctly, survey-based testing arms creative teams with the data they need to focus their early efforts on a small catalog of highly resonant ideas. In these surveys, creative mock-ups are shown to a group of respondents who are asked to react by giving their opinions of the product, brand, or even the particular creative concept. By combining these responses with demographic and psychographic data, we can develop a preliminary view of which assets work best with which audiences.
Survey-based testing provides certain advantages:
- We can safely test out-of-the-box ideas. Since surveys involve a small audience in a low-stakes environment, we can afford to push the boundaries of what a brand would normally do. We often find great concepts that may otherwise be out of bounds.
- We can collect detailed, custom responses. Unlike with live campaign results, surveys tell us how audiences respond to creative assets on an intellectual or emotional level. They let us see beyond click-through or purchase behavior which means we can draw higher-level conclusions about what might resonate and why it does.
- We can analyze audiences deeply. Surveys give us the freedom to collect any audience information we think is relevant. It gives us an opportunity to capture attributes we think may affect purchase behavior and to tie those results to existing market segmentations and buyable attributes that we might want to use when the campaign is live.
- We can test hypotheses beyond the usual key performance indicators. We often work with clients who are interested in “intangible” KPIs like brand trust. Surveys can provide a window into variables that are otherwise very difficult to measure.
But survey-based testing is not a complete solution to creative testing and iteration because of its severe scale, speed, and bias limitations.
Ultimately, survey-based testing should be one piece of a robust testing and optimization framework.
Scale limitations
As with any other scientific trial, we need more survey participants in order to draw stronger conclusions. Ultimately that limits our ability to test a large number of creative ideas or copy. In addition, surveys can be expensive depending on the target audience.
Speed limitations
It takes about a week to field a survey, which is simply not fast enough to iterate on creative concepts several times during a campaign. We need the agility that real-time testing provides to iterate quickly and find the best performing creative assets possible.
Bias limitations
Survey respondents do not necessarily represent the audience who will ultimately see a client’s marketing campaign, and may respond differently to an asset than the target audience would. This disconnect can cause creative teams to make the wrong the wrong call about a campaign’s direction. Sample balancing can help, but the survey-taking population rarely reflects your target market exactly. On top of that, a survey exposes respondents to the creative works out of context. In the wild, an ad competes for attention against various contextual backdrops, which we expect to change campaign effectiveness for better or for worse.
Accuracy limitations
While a survey may help you assess which creative asset does a better job increasing self-reported survey measures like brand awareness, consideration or preference, a survey environment can never exactly match the purchase decision making process.
After all, respondents may not fully understand how they would make an actual purchase decision, and their conscious reflection on their feelings may not accurately match how they would think, decide, and behave in the real world. Clever survey design can control for some of this, but there are limits. Live testing on real-world KPIs like add-to-cart or actual purchase is always valuable for exposing subconscious factors that influence actual performance.
The role of survey-based testing at Known
Because of these bias, speed, and scale limitations, Known is careful to play to survey-based testing’s strengths— it gives us detailed information about how specific audiences react to a small number of well-differentiated concepts. Surveys are a valuable tool when used in this manner, but they are only one piece of a sophisticated testing and optimization suite illustrated below.
By properly leveraging early survey-based concept testing, our talented creative teams can focus their efforts on producing a wide variety of assets around a few resonant concepts. In future posts we’ll talk more about how those assets are leveraged and iterated on to optimize creative assets throughout a full campaign and beyond.
Read more from Known on media buying.
- Want more efficient performance campaigns? Sometimes, less is more
- The importance of data-driven creative
Want to learn about our testing capabilities? Reach out.
