Doug, I love the brilliance of your thesis here. I agree with about half of it. Yes, I’ve seen testing-ad-nauseum misdirect campaigns, and I agree there is a difference between what people say they like and what they actually respond to. Focus groups are as awkward as those overnight sleep apnea studies, where the wires and bright lights can’t help but jar the participants.
However, I’ve also seen creative testing work very well, in both the very early formation stages to validate assumptions, and with in-market testing that tracks actual consumer behavior. Your quote, “It’s a good reminder that they way people respond behaviorally is usually more telling than what they say explicitly. So design for participation, not interrogation…” alludes to this.
- Early on, qual research can break through BOGSAT (Bunch of Guys Sitting Around Table) groupthink errors. For instance, I once worked with a large regional hospital whose management was intent upon rebranding it as a “medical center.” They were sure this would add prestige. Focus groups revealed the opposite: consumers thought of a “hospital” as a large sophisticated facility and a “medical center” as a cheap doctor’s office in a strip mall. That finding helped avoid a multi-million-dollar rebranding mistake for a few years. Eventually the brass got impatient, rebranded awkwardly, and consumers got confused, wondering where their “hospital” had gone. Sigh.
- In-market A-B testing can find solid patterns to redirect campaigns to stronger results. For instance, we worked for three years with Elon Musk’s SolarCity, and did constant A-B testing of radio creative, print, digital, and conversion web pathways at small portions of the media budget before scaling. We often found a 5:1 skew between what worked, and often what worked was the ad creative the management team had not loved in the first place.
I guess my point is, early testing of major themes, and in-market testing of real behavior, can both work well. I guess your point is, creatives shouldn’t allow the fuzziness of over-testing pollute the solid brand or big ideas that will really move the market.
Cheers.
Ben
