Feb 10, 2021
Gauging the Effectiveness of Super Bowl LV Commercials
Using Twitter Data to Quantify Reactions to Various Advertisers
The game was pretty terrible and viewership was way down, but the $5.5 million investment for a Super Bowl commercial may have still been worth it for some companies.
I scraped Twitter using the rtweet package in R to attempt to measure viewers’ reactions to a handful of these advertisements. I selected 12 brands that were reasonably compelling to me, that represented contrasting industries and that used different promotional techniques.
And to be transparent, most of the relevant commercials took place in the first half, where I generally pay much more attention to the ads than in the second half. I’m sure this is not pervasive around the country (or first-half commercials would cost more) but in the second half of the game, I’m either focusing on the football or, in the case of a snooze-fest like this year’s contest, losing focus on the telecast entirely.
I based my data collection on these commercials:
DoorDash; Doritos; Fiverr; Jimmy John’s; Oatly; Paramount+ (1 of 4 ads); State Farm; Tide; T-Mobile; Toyota; Uber Eats; Vroom
Tweet Count
First, I compared the responses to these ads by examining the number of tweets per minute (not including retweets) that mentioned each of these brands over the course of the Super Bowl broadcast.
The responses on twitter make it very evident when the commercials ran for most of the advertisers, including the four separate ads that promoted Partamount+. Unfortunately, I was unable to scrape the Oatly tweets from 6:46–7:38 (which probably not unrelatedly, was the 53 minutes leading up its commercial being shown), but I believe it is safe to surmise that the beginning of its spike would have the same upward slope that the other brands experienced.
Additionally, here is a chart to simply look at the total tweet counts from the entire broadcast:
State Farm achieved the largest Twitter response, as there were more than two times as many tweets in the minute following its commercial than for any of the other advertisers I looked at. Many of the tweets just wanted to make mention of “Drake from State Farm,” which ensured more direct mentions of the brand than, say, the “Flat McConaughey” tweets from the Doritos commercial. While direct mentions likely indicates greater brand awareness, comparisons are imperfect, so both ads seemed to drum up valuable publicity.
On the other end of the spectrum, Jimmy John’s received an exceedingly small Twitter response in its Super Bowl debut.
Despite four separate commercials, Paramount+ could not accumulate the same tweet count as State Farm, Doritos and Oatly. Paramount’s final ad, featuring the SpongeBob “Sweet Victory” song was almost certainly created to evoke a strong social media reaction and it definitely performed the best.
T-Mobile did not have a large spike in response to its commercial and if you simply look at the bar graph to gauge the effectiveness of the ad, you may be misled. T-Mobile actually ran a social campaign during the game:
Thus, use caution when determining how to best use Twitter data to fit a company’s needs. Maybe a commercial provoked a strong reaction but the brand itself was not very salient. To look at this I sought out tweets that mentioned the celebrities involved in the commercials (while not mentioning the brand) in contrast to the tweets that mentioned the brands themselves.
In all two of these commercials, the brand received more mentions than did the corresponding star of each ad. The exception was for Tide and Jason Alexander but Tide already has plenty of brand recognition (and the company’s value was easily understood from the ad). So generating excitement about the inclusion of an actor was probably well within Tide’s goals. Paramount+ also would likely be satisfied with manufacturing attention for Snooki because she will be involved in content on the streaming service. In contrast, engrossment with Big Bird would do little to help DoorDash, so they are probably satisfied with the larger gap between celebrity and brand mentions. Overall, I would hypothesize that all three companies would be satisfied with these results.
Other Forms of Twitter Engagement
The Super Bowl commercials did not appear to encourage engagement through the use of hashtags and the data confirms this. There were only nine different hashtags (or combination of hashtags) that were used at least 200 times (within my subset of tweets) throughout the broadcast.
The use of T-Mobile’s hashtags came through its aforementioned social campaign. Fiverr did the same thing, but earning well over one thousand unique uses of #GoBigWithFiverr is still a formidable outcome. The most impressive result from this chart belongs to Doritos, who briefly flashed the #FlatMatthew hashtag at the end of its commercial. Most of the engagement involving that hashtag was truly organic.
Meanwhile, a new story emerges when you examine retweets and likes.
First, let’s look at the viral tweet that was the driver of that absurd Uber Eats spike:
To add salt to Uber Eats’ wound, here is a tweet that followed five minutes later (which shows up as a little bump on the graph due to the off-the-charts engagement of the prior tweet, but by most standards would still be considered viral):
It seems like the Uber Eats message to “Eat Local” might have backfired as it came off as disingenuous.
The State Farm engagement was not driven by any one viral tweet and thus is a good signal for widespread excitement and amusement over “Drake From State Farm.” The Paramount spike was largely driven by its own tweet of its commercial with SpongeBob, which once again demonstrates how that ad was primed to generate sharing on social media.
Meanwhile, the Oatly spike in likes was not primarily driven by commercial accounts and instead by viral personal takes. Oatly’s 30-second ad featured its CEO singing about the product. It was bizarre, but it successfully provoked a reaction and conveyed that this milk is “meant for humans.”
So it is difficult to deduce meaning by looking at retweets and likes alone. Nuance is required. And considering the varying reasons for engagement, a sentiment analysis is needed to paint a more complete picture.
Twitter Sentiment Analysis
To do this analysis, I used the tidytext package in R, broke down the tweets into individual words and then used the AFINN lexicon to assign numeric values ranging from -5 (most negative) to 5 (most positive).
There was strong positive sentiment for the Fiverr ad, which riffed on the Four Seasons Total Landscaping situation from a few months ago. Celebrity-focused commercials from T-Mobile, Tide and State Farm also generally received a positive reaction. (Although, T-Mobile’s social campaign may bias its sentiment values to some extent.) Toyota was able to appeal to viewers’ emotions with its ad.
On the other hand, Jimmy John’s had the only reaction on Twitter that was more negative than positive. This probably stems from a lack of overall interest in its commercial combined with controversy surrounding its owner.
I’ve mentioned the divisive reactions to Oatly’s commercial, but you can see from the line chart that there was a viscerally negative reaction followed by some more positive tweets in the minutes that followed. And we’ve covered the vitriol towards Uber Eats as well.
Google Trends
Considering the title of this blog post, I should clarify that “success” is dependent on a company’s objectives. And I am certain that many of these brands had different goals in mind and thus will use varying KPIs to measure the value of their commercials. I can take some educated guesses on how such goals may differ.
State Farm, Tide and Toyota are well-established brands and probably care more about customer sentiment than overall engagement levels. Paramount+ and Fiverr are not as well known and probably want to both raise brand awareness but also generate interest in their services. Furthermore, with Paramount+ launching on March 4th, website visits and subscription purchases are probably among its KPI. A relatively unknown company like Vroom is probably more focused on solely generating brand awareness.
These lesser-known brands may also be interested in learning if viewers were searching for more information, which is a great way to measure brand awareness and interest level in the services. I used Google Trends to observe the commercials’ impact on google searches for these companies.
In each graph, the amount of google searches are relative to each other with a value of 100 representing the minute where one of the brands was searched the most. These results reveal that Vroom and Paramount+ derived value from their commercials because viewers were interested in learning more about services that they had likely never heard of before. Most of these other companies were probably less concerned with brand awareness or direct website visits.
As has been apparent throughout this entire post and consistent across every metric, DoorDash won the battle with Uber Eats for food delivery supremacy.
Winners and Losers
So after examining the data and acknowledging that these methods have their limitations, here are my Super Bowl LV winners and losers:
Winners:
State Farm: Tons of engagement; lots of 😍 for Drake from State Farm.
Tide: Very favorable reviews for the inclusion of Jason Alexander.
T-Mobile: Positive reaction, fueled by a successful social campaign.
Fiverr: Highly positive sentiment.
DoorDash: With thanks to Uber Eats’ failures.
Toyota: Favorable reaction from emotional advertisement.
Tampa Bay: The home of professional sports success during pandemics.
And my surprise ultimate winner…
Oatly: The dissenting reactions led to tons of publicity. They also established a clear message (and stuck it in the heads of many). For an ad created six years ago, Oatly was able to accomplish its objectives. And from my perspective, I prefer a commercial that can deliver without any context or celebrities. I never thought there would ever be a time with such a high volume of oat milk tweets nor did I foresee ever devoting so much space in a blog post to oat milk.
Losers:
Uber Eats: Received backlash for purporting to be a companion to local eateries.
Jimmy John’s: Considering the lack of reaction to its ad (which even had star power with Brad Garrett) and how it facilitated unrelated hostility, the sandwich chain may have been better off not making its Super Bowl debut.
CBS: Look at all the classic Super Bowls in the last 15 years. How many appeared on CBS? Maybe 1? (And even still, that was best known for a power outage.)
But there was also one runaway winner that did not have to spend any money, let alone $5.5 million…
What a night for the goat industry! Interest went through the roof during the 10 o’clock hour.
On a related note, seven Super Bowl rings are pretty great for brand image.
Data from Twitter API and Google Trends.
Visualizations created with Tableau.
Check out my github for the code behind each post.
