AI & Pitch Statistics: Learning from Decision Trees and Neural Networks
By: Katherine Lordi, Eli Mauskopf, & Jazmyn Stokes
We implemented a Decision Tree and an Artificial Neural Network to classify pitches as swing or no swing and ball or strike (using 1’s and 0’s) in order to help batters and umpires. Specifically, we used the Python Scikit-Learn package and Python Keras API. In the process, we applied some skills in data science and learned more about these Classification Algorithms. Here’s how we got there!
Our Motivation
In the exploration of theories and methods related to Artificial Intelligence (AI) and Cognitive Science, our team (Katherine and Eli being MLB fans and Jazmyn a D1 softball player) wondered how AI can change the game of baseball due to all of the quantifiable statistics available. For example, sports analytics can provide a team with competitive advantage. How can we use data and AI to help batters improve their game?
“C’mon, Lance!” he shouted from the bench. “It’s the World Series! Wake up!” — Washington Manager Dave Martinez
Additionally, the desire for robot umpires has increased in light of the 2019 World Series. In Game 5, home plate umpire Lance Barksdale called Gerrit Cole’s fastball to Victor Robles as strike three, but the 40,000 in the bleachers did not agree. Robles was upset about the call but noted that “we all make mistakes,” which is part of being human. Umpires have a need for assistance to make accurate calls (especially during high stake games), which is being addressed by new technologies such as TrackMan. How can we use AI to help umpires make more accurate pitch calls?
Our Dataset: MLB Pitch Statistics (2015–2018)
While investigating our motivating questions, we looked for informative datasets with historical baseball statistics. We found a dataset on Kaggle that consists of scraped MLB Pitch Statistics 2015–2018. The dataset provides 40 factors about pitches as columns such as pitch type, outcome, position, velocity, runner placement, spin, break angle, and more. Each row is one pitch, and each year of data has over 700,000 pitches.
The Problem: Pitch Classification
After conducting research and finding MLB Pitch Data for 2015 to 2018, we narrowed our scope to the following classification problems:
- Applications of AI to Help Batters: Determine what kind of ball a batter should swing at in order to increase batting average and decrease strikeouts by classifying pitches as swing or no swing.
- Applications of AI to Help Improve Accuracy of Called Pitches: Help improve the accuracy of called pitches as a ball or strike.
Classification Algorithms Explored
To explore our pitch classification problem, we explored Artificial Neural Networks and Decision Trees, two of the most popular classification algorithms. We also sought to learn about their strengths, weaknesses, and differences. While Artificial Neural Networks (ANN) are biologically-inspired simulations of the neural networks of human brains, Decision Trees create a model that predicts the value of the target variable (leaves) using decision rules and observations (branches).
Neural Networks VS Decision Trees
Black box machine learning models such as Neural Networks are more difficult to interpret and explain while white box models such as Decision Trees have more explainable results by boolean logic. However, there are tradeoffs. Black box models provide for greater accuracy than white box models. Both are flexible algorithms that can handle nonlinear data and interactions between variables, but Decision Trees are more compatible with classification problems while Neural Networks work better with patterns. Additionally, Decision Trees train faster on large amounts of data.
Implementation
We implemented our Decision Tree using the Python Scikit-Learn package and our Artificial Neural Network using the the Python Keras API to solve our pitch classification problem. We have some experience using Keras and wished to apply it to another type of problem.
First, we created dictionaries mapping the code (pitch outcomes) and zone (location of pitch in strike zone) of each pitch to 0 (do NOT swing / ball) and 1 (swing / string). We did this because inputs to the Decision Tree and Neural Network need to be integers. Additionally, it simplifies our categories for classifying pitches to a binary output.
Python Keras Artificial Neural Network
Our Neural Network implementation is inspired by this article, which explains how to construct a Neural Network to classify types of iris flowers. We modified the implementation to consider code and zone as our target outputs and created unique input arrays to specify which inputs to use on a test run. Next, we tweaked the baseline model to have 3 layers with varying input layer dimensions depending on how many columns of pitch statistics we use as input. We used a relu activation function for the hidden layer and a softmax function for the output in order to keep the output score between 0 and 1. For back propagation, we used Adam gradient descent optimization with a logarithmic loss function. To evaluate our model, we used k-fold cross validation, which constructs 10 of our models and evaluates how well they predict our target output. After running our 10-fold cross validation procedure, we printed the average of all folds, standard deviation, and accuracy.
Python Scikit-Learn Decision Tree
Our Decision Tree implementation is inspired by this article, which explains how to to build a Decision Tree Classifier using Python Scikit-learn package. First, we split the MLB Pitch Statistics 2015–2018 dataset into csv files by year using a Python script in order to train our model one year and make predictions on another year. After loading in the pitch data an creating output files, we created the Decision Tree classifier object, trained the Decision Tree classifier, and set the predicted output of the classifier. Finally, we printed the accuracy of the model, which answers the question, “How does the decision tree output to the actual output in the dataset?” For instance, we compare the decision tree making a prediction on code (1 is swing and 0 is no swing) to the actual code column in the dataset.
Results & Analysis
We tried input cases ranging from the pitch’s initial position in the pitcher’s hand to as many relevant pitch statistics as possible. After analyzing the data for input cases we tried, we had some observations:
- Given the same number of input pitch statistics, we cannot determine if the Decision Tree performs better than the Neural Network. This is because we trained the Decision Tree on 100,000 pitches and the Neural Network on 200–1000 pitches due to runtime. Since we did not train the Neural Network on the same amount of data as the Decision Tree, we are not sure which one did better!
- Pitch statistics more related to the pitch’s placement in the strike zone as inputs had higher accuracies for the both the Decision Tree and the Neural Network when predicting code and zone.
- For the Decision Tree, input cases consisting of more pitch statistics had greater accuracy.
Challenges Faced
With some of us being new to the world of AI and data science, we faced some challenges along the way and had to adapt and be creative. One challenge that we faced was the size of the MLB Pitch Statistics dataset, which contaminated over 700,000 pitches per year. While our Decision Tree Implementation ran on all the rows in a matter of seconds, the Keras Neural Network runtime took about four minutes for 1,000 rows of pitch training data. Working with the Keras Neural Network was also challenging since it is a black box model.
Another challenge that we faced was assessing the impact of human error in baseball as reflected in the pitch statistics since batters make mistakes. For example, some pitches that the batter should not have swung at based on its location in the strike zone were actually swung at, which is reflected by the code column. Therefore, we grappled with the accuracy of our Decision Tree and Neural Network and thought about the meaning of the output. Lastly, we had to meticulously consider input cases to train our models and identify meaningful test cases that provide value to batters and umpires.
Conclusions & Reflections
In this project, our team explored AI’s role in sports analytics and baseball, found a dataset of pitch statistics from 2015–2018, and used this dataset to classify pitches as a swing or no swing to help batters and as ball or strike to help improve the accuracy of an umpire’s calls. We implemented a Keras Neural Network and Decision Tree in Python, and through our implementation and results analysis, we saw that AI models are not perfect and are limited by the model setup and the data inputted to them. One future consideration is to add more hidden layers to the Neural Network, which may help improve its accuracy and convergence. Additionally, in the future, we will ensure to train the Neural Network on more pitches to improve its ability to classify pitches given variation of input data.
Therefore, in the realm of baseball, which is played by humans and has so many human factors that can not yet be fully quantified, AI is more likely to aid rather than replace. In baseball, AI is currently helping umpires, and there are no plans to fully replace them. This ties to some exploration of ethical implications of AI and baseball such as AI’s role in the sports industry and sports betting.
See our programs and data here!
