Fantasy Football and The Machine
Using machine learning to get better at fantasy football drafts
There’s an old adage among fantasy football players that says “you can’t win a league in the draft, but you can certainly lose it.” For many who play, the draft is the most anticipated, exciting and enjoyable part of the fantasy football season, but it’s far from a perfect process. Players scour the internet and absorb as much information as possible, including pre-season rankings, projections and of course, average draft position. The problem is none of these methods guarantee you’ll get it right on draft day, so maybe there’s a better way. In this blog post I’ll put a machine learning model to the task of predicting whether or not a player will meet/exceed their average draft position (ADP) ranking.
The goal here is a binary classification — either a player is predicted to meet/exceed ADP rank, or to not meet their ADP rank. For example, consider the case of Saquon Barkley last season. Almost any site you went to had him as the number 1 ranked player in terms of ADP, meaning he was usually the first player drafted. He also had the highest point projection for the season, but ended the year outside the top 5 at his position. If you drafted him 1st overall last year, chances are you suffered for it (I know I did). So how can we get a model to help us make those decisions?
To build this model, I wanted it to have the same kind (and lack) of information a typical fantasy football player would have before the draft: ADP (overall and positional) and projections. In this case, I used projections and ADP for the 2019 season from fantasypros.com. I also needed the capacity to compare how players actually finished the year, so I used 2019 fantasy point totals from fantasypros.com as well. Finally, I included only running backs (RBs), wide receivers (WRs) and tight ends (TEs), referred to as a “flex” group. Quarterbacks, defenses and kickers each have unique scoring, but flex players are all measured by touches (rushing attempts/receptions), touchdowns and yards.
Now that the data was established, the next step was to establish a baseline. I learned that 59% of players last year “met/exceeded” their ADP rank, so if we simply predicted that every player would hit projected value, we’d be right about 60% of the time. As you can see, the problem here is that as you get lower in the draft and the projections get closer to zero, the rate of meeting ADP rank goes up. In a way this makes sense, because if a player is projected to score 0 points but ends up scoring 20 by the end of the year, that player will look like a great value. The reality though is that we really don’t care about those players from a fantasy standpoint (for perspective, Christian McCaffrey led players last year with more than 400 fantasy points scored in PPR leagues).
In light of this, accuracy was clearly not going to be the best metric, and an ROC AUC score would be a better measure of the model’s skill. And in short, an ROC AUC score measures how well a predictive model handles false positives and false negatives. So for example, if the model predicts a player to meet ADP rank, but in fact they do not, this would be considered a false positive and the opposite of this scenario would be considered a false negative. So keep in mind that our baseline for accuracy is 59% and our baseline for ROC is 0.5. The other step here is to split the data into train, validate (val) and test sets, so the model has some of the info to train (learn) on, some of it to validate (quiz) on, and a portion to actually test on. Now let’s get to the modeling.
I used two different models with the first being a Random Forest. For clarity, a Random Forest is a tree-based model that makes predictions based on a series of “decisions.” This model achieved the task of beating the baseline by scoring 64% accuracy on the val set with an ROC score of 0.67.
The second model I used was a Logistic Regression, which works a bit differently than random forest by mapping any real number to an output between 0 and 1 to inform its predictions. While it still had an ROC score of 0.67 for the val set, the accuracy did increase a bit to 66%. However, the reason I chose this model was it’s “low” train accuracy of 67% (random forest had train accuracy well over 80%). This is important because a higher accuracy on the training set indicates a model may overfit the data, and therefore might not perform well on data it has not seen before. So since I want the model to help me make decisions on future fantasy football drafts, I chose logistic regression as it demonstrated a greater ability to generalize.
The final step was to see how this model would perform on the test set I had held out. Ultimately, the logistic regression model scored a test accuracy of 71% and an ROC score of 0.84, both of which substantially beat the baseline. Now before you get too excited about using this on your next draft, I must point out that the sample size was less than ideal at just under 400 observations. However, we can at least say that the model has demonstrated some skill, and I think it serves as a very good starting point that can improve over the years with more data. For now, let’s look into how the model is working and making predictions to see if there’s anything we can learn or start applying ourselves.
We can see that the most important factors the model considered were a player’s positional ADP (pos_adp), projected fantasy points (fpts) and projected touches (att/rec). For any seasoned fantasy football players out there, this may seem pretty obvious. But what was far less obvious and a bit surprising to me was the lack of importance for projected touchdowns. As football fans and fantasy players know, touchdowns are a big deal in real life football and tend to generate the most excitement among fantasy players, so why would the model not consider these events to be “important?” I think this illustrates a broader point which is that any event in sports is difficult to predict, but touchdowns may be one of the most difficult things to predict, and therefore the model simply doesn’t care much about projected touchdowns. This drives home the idea that opportunity is the biggest predictor in fantasy football, and the number of touchdowns a player scores from year to year may be much more noise than signal. Let’s now take a closer look at the model’s predictions in terms of false positives and false negatives.
The confusion matrix illustrates “true” predictions along the diagonal (19 to not hit ADP rank, 21 to hit) and “false” predictions everywhere else (3 false positives, 13 false negatives). I’m quite satisfied that the model only predicted 3 players of the sample to meet ADP rank who actually did not, but I can also recognize room for improvement on the false negatives. For example, the model predicted Zeke Elliott would not meet his ADP rank, when in fact he did.
Ultimately, predicting outcomes in sports is hard, even for computers. The model does a pretty good job of predictions later in the draft, but leaves a bit to be desired in the early rounds. Looking at the image above, we can see why it’s more difficult to predict these players at the top of the draft — only 43% of them hit their ADP rank last year!
Using a partial dependence plot on two features, we can see how the probabilities change based on positional ADP and overall ADP in relation to the target (met ADP rank). As we can see, players at the bottom left of the plot (players at the top of the draft) have the lowest probability of hitting their ADP rank, according to the logistic regression model. Again, this makes some sense because these players have higher expectations, which are naturally more difficult to meet.
The takeaway here is that these early picks are much harder to get right than we might assume. I’ve heard it said before that one doesn’t need to worry much about your first few picks, because these represent all of the best players. This exploration suggests the total opposite, which is that these are the picks you need to be most worried about. So the new lesson for me is to consider a more contrarian approach with picks in the early rounds, as conventional choices are likely to be overvalued.
