50 Million of You Were Part of an A/B Test and Didn’t Even Know It
“If your friends decided to jump off a bridge, would you??” We can still hear the echo of this motherly question from childhood. When we’re young, we often feel the pull of popular opinion so strongly that we do go along with the crowd without question. It’s hard to fight upstream when seemingly everyone is telling us that the crowd knows best. We learn, however, that the crowd isn’t always right.
Companies often feel the same pull of popular opinion: especially when their brand and market share are at stake. Nevertheless, popular opinion on it’s own is far from enough to make decisions on your product, brand and strategy. Testing both internal hypotheses and popular opinion is the only way to ensure the odds are stacked in your favor to win in the marketplace.
Kraft Heinz recently announced what many considered a very risky move: a major change to their macaroni and cheese recipe. The initial reason for the change came from popular opinion as more than 350,000 signatures were attached to an online petition calling for Kraft to remove the artificial colours Yellow 5 and 6 from its cheese sauce.
When it came to making the change to their recipe, Kraft wanted to be sure that when they swapped artificial colours for natural ingredients like paprika, turmeric, and annatto the diehard fans of their product didn’t turn up their collective noses. Kraft Macaroni and Cheese has an incredibly loyal fan base. Their Facebook page alone has almost 1.5 million fans. People even dress up as Macaroni and Cheese for Halloween.
These are not people who want the food they see as a cultural staple to change in any way. Kraft Heinz was concerned that if they told customers that the recipe had changed there would be backlash, whether there was a noticeable taste difference or not.
Kraft decided to test the new recipe, but in a very different way. They actually packaged their new formula and sold it to their customers — 50 million of them. They sold the new formula and saw no drop in sales over a three-month period. They heard no complaints about the taste or colour of this new recipe. This hidden test was a huge success.
When Kraft announced the change after the success of the test, there were some fans who said that they absolutely had been able to taste a difference and voiced their outrage at the change of their favorite product. They were outraged at a change in a product that had gone unnoticed by millions in a truly blind taste test. If Kraft listened to these fans, instead of trusting the tests and data, they would have perhaps made the decision to keep the formula unchanged and had no way to face the trends in the market away from artificial ingredients. The Macaroni and Cheese test saved the day.
The famous neon orange pasta isn’t the only product in Heinz’ portfolio that they’ve felt pressure to change.
In 2010 Heinz faced similar challenges when they felt pressure to reduce the salt contents of their famous ketchup. They didn’t do a secret taste test, but did rely on internally conducted testing on a cross-section of consumers across the country before releasing the newly improved ketchup to the masses.
Many Heinz ketchup lovers were outraged by the decrease of salt in the new recipe. One man in particular was quoted as saying “I’m 80 years old, and I haven’t died yet. It’s really hard for me to eat without salt. I think it’s infringing on our rights!”
Popular opinion in these cases often urges companies to not “fix it if it ain’t broke”. You’ve probably heard the famous quote attributed to Henry Ford: “If I had asked people what they wanted, they would have said faster horses.” It seems even popular opinion in 1915 would lead us to believe that there is never an appetite for change. (Interestingly enough, the public was also duped in this case as it doesn’t appear that Henry Ford ever actually uttered this famous and polarizing phrase.)
Without proper testing (and strong marketing campaigns to announce product changes), would some of our favorite brands be stuck in permanent limbo, unable to ever change their core products?
Many of our clients have faced rough consequences when they have followed popular opinion or hypothesis alone when making changes to their apps. Changes these organizations feel will be great or are backed by anecdotal evidence suddenly look less promising when passed through the rigour of A/B testing.
One example comes from a major retailer. In an attempt to drive better accessibility for their app, they pushed new features to their user base that were thought to be only positive and were left untested. Users seemed to hate the changes as sales dropped significantly directly after the feature changes were pushed to the app. Popular opinion seems to make the most sense, until money is lost.
Relying on only customer feedback, interviews, hypotheses or popular opinion won’t give you a clear view of how your product or changes to a product will fare in the real marketplace. Testing alone also won’t guarantee your success. Using popular opinion and hypotheses as a starting point, however, you can create a solid basis for your tests. A/B testing is the unbiased view you need, to allow you to make the best decision possible in the face of uncertainty.
Going against the crowd can be a real challenge, especially when that crowd is comprised of your loyal customers. The organizations who take the time to test and prove their hypotheses will consistently win. These time-tested brands will continue to win over even the incredulous few diehard fans who will swear they can tell the difference in the new product and even demand a return to their old familiar favorite.
Taplytics is a fully integrated mobile A/B testing, push notification, and analytics platform providing the tools you need to optimize your mobile app.
