Hey everyone, we beat ESPN — Part 2. Results

Comparison of fantasy football weekly model performance of Fantasy Outliers vs. ESPN during Weeks 6–16 of the 2017 NFL season for Standard and PPR scoring formats

9 min readJul 31, 2018

--

Introduction (copied from Part 1: Methodology)

In 2015, Fantasy Outliers started as a bunch of braggy graphs that I made when I won a league. Since then, we’ve made historical interactive graphs where users can explore our value-based analysis of what actually happened in competitive leagues, yearly predictive models to help with the draft, and weekly predictive models that help determine whom to start/sit and which under-the-radar free agents are worth their salt.

Last year, we beta-tested Fantasy Outliers’ predictive models, affectionately referred to from here on out as MathBox. Anecdotal results were as follows:

Though a small sample size, a disproportionate number of us won our leagues last year. Notably, in the league that birthed Fantasy Outliers, someone who had historically finished in the bottom three of our league finally won — suggesting that MathBox ‘leveled the playing field’, so to speak.
We found that by predicting performance in the upcoming four games and through the rest of the season, MathBox helped us spot Free Agent pickups a week or two before everyone started talking about them.
Finally, pretty much every single time last year when setting my starting lineups, I made good decisions when using a combination of fantasy football knowledge and MathBox’s predictions and bad decisions when using solely ESPN’s predictions.

But this is anecdotal and based on a small sample size, so we wanted to put it to the test. Was MathBox actually better than ESPN or not? If so, what was it good or bad at? What are the best and worst ways to use MathBox’s predictions to gain a competitive advantage when making decisions for your team?

For the remainder of the article, we will cover the Results in greater detail. For Executive Summary and Methodology of how our weekly predictions were created and compared to ESPN’s weekly predictions, visit Part 1.

Results — Comparing Fantasy Outliers’ vs. ESPN’s weekly predictions

Quarterback

Quarterback was clearly MathBox’s best performing position. This makes sense, because there is more consistency to the quarterback position in the NFL, more teamwide and opponent trends are directly relevant, and QBs get more opportunities per game, thus reducing the volatility in their scores relative to other positions.

MathBox performed better for top-ranked QBs than low-ranked QBs, and was an exceptional directional indicator relative to ESPN.

QB Points: For quarterbacks rated in the top 10 by ESPN going into the week, MathBox’s straight-up points projections outperformed ESPN’s by 56W-42L for an approximate 7.8% chance that ESPN’s models were better. For top 20-rated QBs, MathBox’s predictions went 112W-91L for a 6.9% chance that ESPN’s models were better, but for QB’s rated 20–30 going into the week, it was a wash. This is without accounting for situational anomalies, where ESPN has a built-in advantage since their models were hand-adjusted. Also, there were a lot of back-ups that started in 2017, and maybe MathBox just didn’t have enough data on them yet to get into full gear.
QB Increment ESPN directionally: When thinking of MathBox’s projections as directional indicators relative to ESPN, rather than raw points projects, we see in uptick in performance with MathBox coming in at 62W-36L for Top 10 QBs at a 63.3% ‘winning rate’ (Note this is not a winning rate vs. ESPN, but rather a winning rate versus whether the player is going to score more or less points than ESPN’s prediction). For Top 30 QBs, MathBox went 178W-121L for near zero percent odds that the results are due to chance.
Individuals: The quarterbacks that MathBox performed the best on were Case Keenum, Philip Rivers, and Blake Bortles at 9W-1L each. I love the Case Keenum and Blake Bortles examples especially, because they are ‘Moneyball-esque’ (underrated players). Notably, MathBox weeklies went 8W-2L for Russell Wilson, while the yearly models going into the season had him at #1 (he finished the year at #1) — go MathBox!

**QB model comparison**: For Points projections, numbers represent the probability that ESPN weekly models are better than MathBox’s weekly models. For *Increment ESPN directionally* comparisons, the numbers represent the probability that MathBox’s directional performance is worse than chance. Low numbers (dark blue) are better for MathBox.

Running Back

Running backs (RBs) are a notoriously difficult position to predict in the NFL. The high injury rates doesn’t make life any easier for fantasy — and real — football gurus alike. ESPN mostly beat out MathBox in this position group — likely due, at least in part, to the higher degree of situational hand-adjustments required — but there were some notable exceptions.

MathBox performed on par with Top 10 RBs, best at mid-tier/top 10–20 RBs, worst at lower-tier/20+ RBs, slightly better in PPR scoring, and was directionally relative to ESPN very effective.

Points: For Top 10 RBs, MathBox and ESPN were statistically 50/50 — about equal. In our opinion, this is a win for MathBox, because we think that if one was to make strategic hand adjustments going into the week, MathBox would have won at least a few more match-ups. For Top 10–20 RBs, MathBox went 41W-33L in Standard and 43W–36L in PPR — nothing to write home about. ESPN crushed MathBox for RBs rated 21 or higher going into the week with MathBox’s performance at 154W-281L and 210W-338L in Standard and PPR formats, respectively.
Increment ESPN directionally: While MathBox may not provide a clear, across-the-board advantage for RBs in terms of raw point projections as it seems to do for QBs, its directional performance relative to ESPN remains strong. For Top 20 RBs, MathBox’s record was 98W-73L in Standard and 95W-81L in PPR formats or 2.8% and 14.1% odds, respectively, that the results are due to random chance.
Teams: MathBox’s best performing team in terms of Running Backs? New England. Yes, you heard that right. MathBox’s Opportunities*Points Per Opportunity and Directional models scored somewhere between 20–23W to 7–11L for NE RBs in Standard and PPR leagues — while MathBox’s raw Points model lagged behind at 19W-14L. Did MathBox, in its 6K+ columns, unravel a small piece of the Bill Belichick/Tom Brady running back mystery that has plagued fantasy football noobs and gurus alike for years? Only time will tell. Let’s see if these results hold up in 2018.
Individuals: MathBox beat out ESPN in all formats for Alvin Kamara at 9W-2L. MathBox came in at 8W-2L for most models for LeSean McCoy (Buf), Dion Lewis (NE), and Javarius Allen (Bal).

**RB model comparison**: For Points projections, numbers represent the probability that ESPN weekly models are better than MathBox’s weekly models. For *Increment ESPN directionally* comparisons, the numbers represent the probability that MathBox’s directional performance is worse than chance. Low numbers (dark blue) are better for MathBox.

Wide Receiver

Similar to RBs, MathBox’s WR performance was much better in PPR than Standard scoring, an uptick in performance in the mid-tier/Top 10–20 range, and was better when used as a directional indicator.

Points: For PPR, Top 30 WRs, MathBox’s raw points performance trended positive compared to ESPN, but it was close at 106W-93L in the Top 20 and 149W-140L in the Top 30, respectively. In our opinion, this is a ‘win’ for MathBox, because ESPN had the inherent advantage of making hand-adjustments for situation changes. For Standard scoring, MathBox was pretty dead even to ESPN in the Top 20 — but for 20+ WRs, MathBox got crushed at 237W-356L.
Increment ESPN directionally: When using MathBox as a directional indicator, performance improves, but the story stays the same: better in PPR at the Top 20 WRs.
Teams: MathBox’s best performing WR group was San Francisco at 20W-7L in raw points PPR comparisons. Other teams near the top in the MathBox’s win column were Ten, NYG, Jax, LAR — not teams known for their wideouts (especially, last year with OBJ out).
Individuals: Davante Adams, Devin Funchess, Jarvis Landry, DeAndre Hopkins all topped the list of MathBox victories at about 8W to 1–2L in PPR raw points predictions. Names like Keenan Allen, Sammy Watkins, and Jamison Crowder came in a close second at about 7W-3L in PPR. Interestingly, in most cases, these WRs weren’t flashy, media-hyped kind of WRs last year.

**WR model comparison**: For Points projections, numbers represent the probability that ESPN weekly models are better than MathBox’s weekly models. For *Increment ESPN directionally* comparisons, the numbers represent the probability that MathBox’s directional performance is worse than chance. Low numbers (dark blue) are better for MathBox.

Tight End

MathBox’s performance for Tight Ends followed a similar pattern to that of RBs and WRs — better PPR performance, better performance in the Top 20 tier, and more success as a directional indicator.

Points: MathBox fared much better in PPR than Standard scoring for TEs. MathBox’s PPR Top 10 record came to 55W-44L, Top 20 at 100W-91L, and Top 30 at 132W-136L. We’re calling that a win for MathBox, given lack of situational adjustments in MathBox’s predictions. But for Standard scoring leagues, ESPN dominated the TE raw points predictions with MathBox losing 121W-184L.
Increment ESPN directionally: When used as directional indicators relative to ESPN’s projections, MathBox came in at 109W-82L for Top 20 TEs in PPR for 2.5% odds the results are due to chance along. For Top 20 TEs in Standard scoring, ESPN was the victor with a 91W-111L record for MathBox.
Teams: MathBox’s best teams in terms of TE performance were Sea, Jax, and Ind, coming in at 18W-4L, 16W-8L, 15W-4L for the directional models
Individuals: Jack Doyle (Ind), Jimmy Graham (Sea), and Eric Ebron (Det) all came in at 9W-1L for both the points and directional models, and Tyler Kroft (Cin) was a notable 8W-2L. Again, at a glance, these appear to be somewhat under-the-radar players.

**TE model comparison**: For Points projections, numbers represent the probability that ESPN weekly models are better than MathBox’s weekly models. For *Increment ESPN directionally* comparisons, the numbers represent the probability that MathBox’s directional performance is worse than chance. Low numbers (dark blue) are better for MathBox.

Conclusion

These results are encouraging to those of us at Fantasy Outliers. We are a small team, we used a poor man’s dataset of traditional statistics, worked on our projections in our ‘free’ time, and as stated numerous times above, were not able to make backward-looking hand-adjustments based on scenarios not accounted for in our dataset. We were comparing to models made a behemoth corporation in ESPN that DID have the advantage of using situational hand adjustments and incorporated expert knowledge.

The results tended to show that our models were better for QBs, PPR formats, Top 20 players, and a better directional indicator than raw points estimate. In some instances, we were on par with ESPN without human adjustments, and in other cases, we performed much better. Of course, ESPN had their wins, too, mostly in Standard formats and players ranked 20th on upwards. It will be interesting how these results change after the 2018 where we plan to record our human/expert adjustments to the models and make some improvements to our modeling process.

As we expected, MathBox’s Points models generally outperformed its Opportunities * Points Per Opportunities models. This suggests the notion that the Opp*PtsPerOpp models are best used as a diagnostic tool when making hand adjustments the Points models — in other words, if you disagree with the number of projected opportunities, just change that value and see where the resulting Opp*PtsPerOpp model lands.

Future research could look into more advanced datasets, different modeling techniques, Kickers and Defensive projections, comparisons to a wider array of projections, keeping record of hand-adjusted projections, applying our underlying techniques to different fantasy sports, and exploring how our techniques can help people make decision in not just fantasy, but real sports, too.

Please, join us in our quest to bring data science to the fantasy football world. Follow us on Medium, follow us on Twitter, and together, let’s dominate our fantasy football leagues in 2018!

If you haven’t already, visit Part 1: Background and Methodology.