How Artificial Intelligence (AI) beat ESPN in Fantasy Football
Summary of results of Fantasy Outliers’ weekly predictive models vs. ESPN during Weeks 6–16 of the 2017 NFL regular season
View our product line (fantasy league, DFS, and NFL betting tools) here
Don’t let the title fool you. That’s why we put our slogan right below it. We’re not suggesting that Artificial Intelligence is going to replace football decision-making anytime soon. Our core belief is:
Human expertise combined with data science is better than either by itself.
Experts and computers simply have access to different information, and when combined strategically, they can do great things together. Centaurs will rule the world!
- Experts can see someone’s explosiveness on tape; get a feel for someone’s personality and drive; assess their technique; talk to coaches, players, and other pundits
- Computers can search through thousands of columns across millions of rows and find the best, most historically predictive combination of statistics for a given player’s history
Although, there is definitely some overlap between the two, generally speaking, the strengths of experts and computers are symbiotic.
With that in mind, in this comparison, ESPN’s projections used both computers and experts, whereas Fantasy Outliers’ projections only used computers — so ESPN had an inherent advantage. This was because we finished tweaking our models in the offseason, and it would be unfair to retroactively adjust them for situational anomalies. So, for example, if the starting RB tore his ACL the previous week, ESPN’s projections would take this into account right away, whereas it would probably take our models a week or so to figure it out.
In other words, we’re comparing our models’ performance to ESPN’s with one arm tied behind our back. Spoiler alert: we still beat them in many ways!
Even with this built-in handicap, our models still were either on par or better than ESPN’s in many (not all) comparisons.
Anecdotal results
Last year, we beta-tested Fantasy Outliers’ predictive models, affectionately referred to from here on out as MathBox. Anecdotal results were as follows:
- Though a small sample size, a disproportionate number of us won our leagues last year. Notably, in the league that birthed Fantasy Outliers, someone who had historically finished in the bottom three of our league finally won — suggesting that MathBox ‘leveled the playing field’, so to speak.
- We found that by predicting performance in the upcoming four games and through the rest of the season, MathBox helped us spot Free Agent pickups a week or two before everyone started talking about them.
- Finally, pretty much every single time last year when setting my starting lineups, I made good decisions when using a combination of fantasy football knowledge and MathBox’s predictions and bad decisions when using solely ESPN’s predictions.
But this is anecdotal and based on a small sample size, so we wanted to put it to the test. Was MathBox actually better than ESPN or not? If so, what was it good or bad at? What are the best and worst ways to use MathBox’s predictions to gain a competitive advantage when making decisions for your team?
How did we make our predictions?
If you are a geek like us and want a more complete description, you can read Part 1. Background and Methodology. That said, here’s the gist:
- We used a dataset of traditional, non-fancy NFL statistics, obtained both via the public web and armchairanalysis.com
- We created nearly seven thousand columns related to individual and team, offense and defensive stats. We limited the dataset to up through Week 5 of 2017, so that we could test performance on Weeks 6–16
- We then ran each combination of position (QB, RB, WR, TE) and scoring format (Standard and PPR) through a pipeline of data processing and machine learning techniques that finds the best, most predictive combination of settings and columns for the model
In short, instead of analyzing fancy NFL statistics in traditional ways, let’s just say we analyzed traditional NFL statistics in fancy ways.
We finished tweaking our models during the offseason, so we did not have the benefit of looking backward to make hand adjustments each week for situational anomalies.
How did we compare our predictions to ESPN’s?
We obtained ESPN’s predictions by essentially copying them off the web. Then for each position, scoring format, and week we rated which projection (Fantasy Outliers’ or ESPN’s) was closer to what actually happened. We then counted up these ‘wins and losses’ throughout weeks 6–16 to make our comparisons.
We also counted up the number of times Fantasy Outliers’ predictions were directionally accurate relative to ESPN’s projections. In other words, if our prediction was higher than ESPN’s and the player scored higher than ESPN’s, we scored it as a win; if they scored lower, it was a loss.
So what were the results?
The following is a summary of results. For a more detailed look, visit Part 2: Results.
Things we did well on…
- Quarterbacks: Our best position. Maybe it’s because Quarterbacks have more attempts per game which evens out some of the volatility or maybe the inner workings of the team dynamics are more relevant. Either way, we outperformed ESPN significantly for both points projections and when used as directional indicators. As you can see from the following chart our win rate vs. ESPN, we were most accurate for higher rated QBs, but provided value for all tiers:
- Top 20 Running Backs and Wide Receivers: For points comparisons, we were either on par (Top 10) or better (Top 10–20) than ESPN’s projections. Keep in mind that this is without the benefit of making hand adjustments for situations that aren’t accounted for in the dataset — like the starter went down last week. As a directional indicator, we did even better.
- Point projections in PPR: Maybe it’s due to the fact that there is more predictability in PPR scoring, where you get one point per reception. But whatever it is, our PPR points projections tended to outperform our points projections in Standard scoring leagues
- Directional Indicator: When used as a directional guide relative to ESPN’s projections our models were very helpful. In fact, for typical fantasy football ‘starters’ — that is, Top 10 QB’s and TE’s and Top 20 RB’s and WR’s — our models performed better than chance for both Standard and PPR scoring formats. Here, you can see our winning percentage when used as a directional indicator for likely fantasy football starters who have played in at least two consecutive games. (By the way, these results — when combined across all positions — are probabilistically significant at 99.8% confidence in Standard 99.99% confidence in PPR.)
- New England running backs: Historically, running backs for the New England Patriots and fantasy football have not played well together. Coach Bill Belichick always keeps us wannabes guessing. Our models might have cracked at least some of the code, since our models clearly did better on NE RB’s than any other team — relative to ESPN, that is.
- Under-the-radar individuals: Last year, we felt that our models were consistently accurate for certain players more than others. Some under-the-radar players for whom our predictions were the very accurate include: Case Keenum, Philip Rivers, and Blake Bortles for QB; Dion Lewis, Javarius Allen for RB; Davante Adams, Devin Funchess, Jarvis Landry, Sammy Watkins, Jamison Crowder for WR; Jack Doyle, Tyler Kroft, and Eric Ebron for TE. Some over/under-valued stars we predicted well, include: Russell Wilson, Alvin Kamara, LeSean McCoy, Keenan Allen, DeAndre Hopkins, and Jimmy Graham.
- Likely starters: If we consider 1QB, 2RB, 2WR, 1TE leagues with 10-Teams, then it is reasonable to define likely starters to be Top 10 QB and TE, and Top 20 RB and WR. With this in mind…
…our point projections were on par or better than ESPN’s for likely starters all positions and scoring formats except TE’s in Standard scoring.
Things we did poorly on…
- Points projections for bottom 20+ running backs and wide receivers: Whether it’s due to the increased volatility/touchdown-dependent nature of this group, reduced historical dataset in some cases, lack of hand adjustments, or something fixable in our models (that’s my guess) ESPN generally beat us in this category. With that said, our models still were still useful as a directional indicator in this group.
- Points projections for bottom 10+ tight ends: Same story here. The points projections were less accurate than ESPN’s but the models still provided some directional value.
- Certain individuals: A random selection of players we did not predict well last year, relative to ESPN, include: Jordy Nelson, LeGarrette Blount, Jay Cutler, James Conner, Ameer Abdullah, Charcandrick West, Jamaal Charles, Shane Vereen, DeAndre Washington, Amari Cooper, Cole Beasley, J.J. Nelson.
How can you use these models going forward?
There is a lot of info in this article that will help us know when to listen to the models and when to ignore them; when to use them only as directional indicators, and when to also trust their point projections.
All in all, we were pleased with these results, but there is lots more research to be done. It kinda feels like this is just the tip of the iceberg as far as what data science can do for fantasy football — or even the real NFL.
Keep in mind, that if we had made hand adjustments only for obvious situational anomalies, the results would likely have been at least a few percentage points better across the board.
To stay in touch going forward, please follow us on twitter, sign up for our weekly newsletter (weekly during the NFL season, that is), or view our product line (fantasy league, DFS, and NFL betting tools) here.
If you’re curious and want to learn more. Here are the full results of our weekly model comparison:
Also, here are the results of our yearly model comparison: Can machine learning help your fantasy football draft?
For predicting NFL game winners, there’s this: We Tied Vegas in Our First Attempt at Predicting NFL Game Winners Using Machine Learning.
Ray Harris contributed to this article.
