Super Bowl Data Analysis with Python
All 54 Super Bowls for Fans and Non-Fans, in a nutshell!
As a person who likes sports, I decided to get my hands on this project to apply my skills in python and data science. I really like Football, but I do not have a whole understanding of its past and some rules as well. So, in this project I am answering some questions I have made myself, questions from you (why not), and other questions that are part of the project. The project is one of the multiple projects that DataCamp offers in its “Data Scientist with Python” track. The original scope of the project considered only 52 Super Bowls through 2018, but I added more data for Super Bowl XLIII and XLIV.
1. TV, halftime shows, and the Big Game
Whether or not you like football, the Super Bowl is a spectacle. There’s a little something for everyone at your Super Bowl party. Drama in the form of blowouts, comebacks, and controversy for the sports fan. There are the ridiculously expensive ads, some hilarious, others gut-wrenching, thought-provoking, and weird. The half-time shows with the biggest musicians in the world, sometimes riding giant mechanical tigers or leaping from the roof of the stadium. It’s a show, baby. And in this notebook, we’re going to find out how some of the elements of this show interact with each other. After exploring and cleaning our data a little, we’re going to answer questions like:
- What are the most extreme game outcomes?
- How does the game affect television viewership?
- How have viewership, TV ratings, and ad cost evolved over time?
- Who are the most prolific musicians in terms of halftime show performances?
- What are the top games with more viewership?
- What are the most rated Super Bowls?
The dataset we’ll use was scraped and polished from Wikipedia. It is made up of three CSV files, one with game data, one with TV data, and one with halftime musician data for all 54 Super Bowls through 2020. Let’s take a look, using display() instead of print() since its output is prettier in Jupyter Notebooks.
2. Taking note of dataset issues
For the Super Bowl game data, we can see the dataset appears whole except for missing values in the backup quarterback columns (qb_winner_2 and qb_loser_2), which make sense given most starting QBs in the Super Bowl (qb_winner_1 and qb_loser_1) play the entire game.
From the visual inspection of TV and halftime musicians data, there is only one missing value displayed, but I’ve got a hunch there are more. The Super Bowl goes all the way back to 1967, and the more granular columns (e.g. the number of songs for halftime musicians) probably weren’t tracked reliably over time. Wikipedia is great but not perfect.
An inspection of the .info() output for super_bowls, tv, and halftime_musicians shows us that there are multiple columns with null values.
Besides the information of the columns, the output shows us the different kind of data type and the total of columns with each data type. If you want to know more about data types, you can read my previous article:
3. Visual inspection of missing values
It is said that a picture is worth a thousand words, so I like looking at some visuals. In this case, we can confirm those dataset issues with a missing values heatmap. Let’s use yellow for our missing values and blue for non-missing values.
Super Bowls dataset heatmap:
TV dataset heatmap:
Halftime Musicians dataset heatmap:
4. Combined points distribution
For the TV data, the following columns have missing values and a lot of them:
total_us_viewers(amount of U.S. viewers who watched at least some part of the broadcast)rating_18_49(average % of U.S. adults 18-49 who live in a household with a TV that were watching for the entire broadcast)share_18_49(average % of U.S. adults 18-49 who live in a household with a TV in use that were watching for the entire broadcast)
For the halftime musician data, there are missing numbers of songs performed (num_songs) for about a third of the performances.
There are a lot of potential reasons for these missing values. Was the data ever tracked? Was it lost in history? Is the research effort to make this data whole worth it? Maybe. Watching every Super Bowl halftime show to get song counts would be pretty fun. But we don’t have the time to do that kind of stuff now! Let’s take note of where the dataset isn’t perfect and start uncovering some insights.
Let’s start by looking at combined points (final score combining points from both teams) for each Super Bowl by visualizing the distribution. Let’s also pinpoint the Super Bowls with the highest and lowest scores.
5. Point difference distribution
Most combined scores are around 40–50 points, with the extremes being roughly equal distance away in opposite directions. Going up to the highest combined scores at 74 and 75, we find two games featuring dominant quarterback performances. One even happened recently in 2018’s Super Bowl LII where Tom Brady’s Patriots lost to Nick Foles’ underdog Eagles 41–33 for a combined score of 74.
Going down to the lowest combined scores, we had in Super Bowl LIII in 2019 the Patriots and Rams braking the record for the lowest-scoring Super Bowl with a combined score of 16 points (13–3 for the Patriots); the game was led by the great defense of both teams. Then, we have Super Bowl III and VII, which featured tough defenses that dominated. We also have Super Bowl IX in New Orleans in 1975, whose 16–6 score can be attributed to inclement weather. The field was slick from overnight rain, and it was cold at 46 °F (8 °C), making it hard for the Steelers and Vikings to do much offensively. This was the second-coldest Super Bowl ever and the last to be played in inclement weather for over 30 years. The NFL realized people like points, I guess.
Let’s take a look at point difference now.
6. Ratings distribution
The vast majority of Super Bowls are close games. Makes sense. Both teams are likely to be deserving if they’ve made it this far. The closest game ever was when the Buffalo Bills lost to the New York Giants by 1 point in 1991, which was best remembered for Scott Norwood’s last-second missed field goal attempt that went wide right, kicking off four Bills Super Bowl losses in a row. Poor Scott. The biggest point discrepancy ever was 45 points (!) where Hall of Famer Joe Montana’s led the San Francisco 49ers to victory in 1990, one year before the closest game ever.
Is it possible that New York Giants vs Bufallo Bills game in Super Bowl XXV had the best rating in history? or are the games with more combined points in Super Bowl LII and XXIX the best rated? We can take a look at the distribution of rating as well as check on the relationship between ratings and difference in points/combined points.
7. Do blowouts translate to lost viewers?
Our scatter plot in blue has almost no slope in our regression line indicating there is not relationship between having the difference of points and ratings. On the other hand, you can see there is a relationship between combined points and ratings in the green scatter plot which makes sense because a game with plenty points is a good game. We can see in our histogram that most of the Super Bowls had ratings between 40-50. There is only one game that is shown in our display, and it is Super Bowl XX which had 36 points of difference, 56 points in combination during the game, and a 48.3 rating. Our tightest game in Super Bowl XXV with 1 point of difference is not even in our display achieving just 41.3 of rating. In fact, Super Bowl XVI had the best rating of history with a value of 49.1, well done!
I remember watching the Seahawks crush the Broncos by 35 points (43–8) in 2014 (sorry Cesar), which was a boring experience in my opinion. The game was never really close. I’m pretty sure we changed the channel at the end of the third quarter. Let’s combine our game data and TV to see if this is a universal phenomenon. Do large point differences translate to lost viewers? We can plot household share (average percentage of U.S. households with a TV in use that were watching for the entire broadcast) vs. point difference to find out.
8. Viewership and the ad industry over time
The downward sloping regression line and the 95% confidence interval for that regression suggest that bailing on the game if it is a blowout is common. Though it matches our intuition, we must take it with a grain of salt because the linear relationship in the data is weak due to our small sample size of 52 games.
Regardless of the score though, I bet most people stick it out for the halftime show, which is good news for the TV networks and advertisers. A 30-second spot costs a pretty $5 million now, but has it always been that way? And how have number of viewers and household ratings trended alongside ad cost? We can find out using line plots that share a “Super Bowl” x-axis.
9. The Super Bowl with more viewership
We can see viewers increased before ad costs did. Maybe the networks weren’t very data savvy and were slow to react? Makes sense since there were few data analytics and data science methods before the 2000s.
Viewership increases overtime, but this is not always the case as you can see in the past few years in the average number of US viewers line graph. Does this has to be because new generations prefer streaming media platforms over networks? Which is the Super Bowl with more viewership? I thought it had to be with Super Bowl XXVII where Michael Jackson performed a fantastic show in 1993, but the line graph tells something different. Well, I still think it is the best show ever!
Another decade, another Jackson, though this time the wardrobe remained malfunction free. Michael Jackson’s 1993 appearance at the Rose Bowl in Pasadena, California was a turning point in halftime extravaganzas, becoming one of the most-watched events in US history. — Stephen Dowling, “10 best Super Bowl Halftime shows of all time”, BBC Music, 31st January 2020.
10. Halftime shows weren’t always this great
It turns out that Super Bowl XLIX was the one with more viewership. How not to forget Katy Perry’s left shark?
Another hypothesis: maybe halftime shows weren’t that good in the earlier years? The modern spectacle of the Super Bowl has a lot to do with the cultural prestige of big halftime acts. I went down a YouTube rabbit hole and it turns out the old ones weren’t up to today’s standards. Some offenders:
- Super Bowl XXVI in 1992: A Frosty The Snowman rap performed by children.
- Super Bowl XXIII in 1989: An Elvis impersonator that did magic tricks and didn’t even sing one Elvis song.
- Super Bowl XXI in 1987: Tap dancing ponies. (Okay, that’s pretty awesome actually.)
From our Ad Cost line graph, you can see that Michael Jackson's Super Bowl XXVII performance, one of the most watched events in American TV history, was when the NFL realized the value of Super Bowl airtime and decided they needed to sign big name acts from then on out. The halftime shows before MJ indeed weren't that impressive, which we can see by filtering our halftime_musician data.
11. Who has the most halftime show appearances?
Lots of marching bands. American jazz clarinetist Pete Fountain. Miss Texas 1973 playing a violin. Nothing against those performers, they’re just simply not Beyoncé. To be fair, no one is.
Let’s see all of the musicians that have done more than one halftime show, including their performance counts.
12. Who performed the most songs in a halftime show?
The world famous Grambling State University Tiger Marching Band takes the crown with six appearances. Beyoncé, Justin Timberlake, Nelly, and Bruno Mars are the only post-Y2K musicians with multiple appearances (two each).
From our previous inspections, the num_songs column has lots of missing values:
- A lot of the marching bands don’t have
num_songsentries. - For non-marching bands, missing data starts occurring at Super Bowl XX.
Let’s filter out marching bands by filtering out musicians with the word “Marching” in them and the word “Spirit” (a common naming convention for marching bands is “Spirit of [something]”). Then we’ll filter for Super Bowls after Super Bowl XX to address the missing data issue, then let’s see who has the most number of songs.
So most non-band musicians do 1–3 songs per halftime show. It’s important to note that the duration of the halftime show is fixed (roughly 12 minutes) so songs per performance is more a measure of how many hit songs you have. JT went off in 2018, wow. 11 songs! Diana Ross comes in second with 10 in her medley in 1996.
13. Conclusion
In this project, we loaded, cleaned, then explored Super Bowl game, television, and halftime show data. We visualized the distributions of combined points, point differences, viewership, and halftime show performances using histograms. We used line plots to see how ad cost increases lagged behind viewership increases. And we discovered that blowouts do appear to lead to a drop in viewers, and there are better ratings when games have high scores.
This year looks uncertain with COVID-19 and the economy; perhaps, we should work on predictive analytics to forecast the future with COVID-19 and the next Super Bowls? That’s a task for another moment.
So, tell me, do you have any Super Bowl question that you want me to get from data?
About me
Hi, I am Raul. Thank you so much for reading my article! I am passionate about Data Science and Data Analytics applications and their impact to the world. I started publishing as part of my continuous learning because this is one of the best ways for me to stick things in my mind. If you have questions, please don’t hesitate to contact me!
Get in touch at hrcontre@asu.edu and stay connected on LinkedIn. Visit my Personal Website and GitHub
