An In-Depth Look Into Collegiate Catcher Framing

Introduction

Catcher framing has been studied extensively in Major League Baseball for the past several years, but has yet to be fully explored in the realm of college baseball. This concept of framing is incredibly valuable for the pitching team, especially in high-leverage situations, as it can be the difference between winning and losing a game. By stealing strikes, a hitter’s count turns into a pitcher’s advantage, or a would-be walk becomes a strikeout. Because of this, the purpose of this project was to create a system to evaluate a catcher’s ability to keep strikes as strikes and convert balls to strikes in the college game.

Last year, Christian Hook wrote an article valuing catcher framing at the Major League level. Hook’s research served as a great launching point for the initial stages of this project. In his article, catchers were assigned credit for strikes and balls based on a called strike probability model and the change in run expectancy. The catcher framing metric that I developed for Iowa shares a similar methodology, but is set up differently due to the differences in available data and high talent fluctuation at the collegiate level.

Measuring Responsibility

An incredibly important factor in evaluating a catcher’s framing skills is how much they should be rewarded or penalized for certain pitches. If a pitcher throws a pitch right down the middle, the catcher did not need to do much to convince the umpire it was a strike. On the other hand, a pitch spiked in the dirt is out of the catcher’s control. In between these two extremes are the edges of the strike zone. This area is a toss-up as to whether or not the pitch will be called a strike or called a ball. In the past, pitchers were commended for “painting the black”, however, the catchers have played a significant role in persuading the umpire to call those pitches strikes. To control what a catcher has responsibility for, this analysis starts by looking at called strike probability.

Last Fall, our analytics department created a system to quantify swing decisions in hitters. One piece to the “swing decision score” was the creation of a called strike probability model. Since we published that article, the called strike probability model was updated to increase the accuracy.

For the current version, k-fold cross-validation was used where our Trackman dataset was randomly broken up into four equally-sized folds. The folds were then filtered to consist of only called strikes and called balls. Three of the folds were used as training data and the remaining fold was used for testing. The training data was put through the Random Forest method. This process was repeated four times so each fold could be tested. The following variables were included in the model:

• Vertical Pitch Location
• Horizontal Pitch Location
• Batter Handedness
• Count
• Run Differential (Pitching Team)

Often times during the Fall, teams will scrimmage without an umpire and use Trackman as a means for calling balls and strikes. Therefore, any games that did not involve a home plate umpire were removed from the training set. After testing every fold, they were combined to train the non-umpire scrimmage data. Based on a confusion matrix, the model was 88.6% accurate. Due to the wide talent range in umpires, it makes it tough to have an accuracy in the high 90’s.

To quantify the impact of each pitch on the game, the run expectancy matrix, RE144, was formulated. This matrix contains outs, count, and the number of runners on base for a total of 144 unique situations. Since this project uses data from Trackman, it was not possible to determine the exact bases occupied at any given pitch or plate appearance. This is something that the technology does not track during the game or possess in its CSV counterpart. Alternatively, this RE144 matrix uses the estimated number of runners on base, which was calculated with data manipulation techniques in R. The understanding of the game situation plays a key role in developing one of the metrics in the following section.

Framing Metrics

The final system for valuing catcher framing came down to two different metrics — one with context and one without. Not all situations are created equally in a baseball game, so there is significant value in a catcher’s ability to steal a strike in a high-leverage situation. These two statistics — Contextual Catcher Runs and CatchCS — allow us to analyze how the catcher performs under pressure and understand his overall framing ability, respectively.

Contextual Catcher Runs (CCR)

For this statistic — named Contextual Catcher Runs — the called strike probability of a pitch was combined with the change in RE144. The formula for Contextual Catcher Runs for a pitch is as follows:

For a strike: CCR = (1-Called Strike Probability) * change in RE144 * -1

For a ball: CCR = Called Strike Probability * change in RE144 * -1

To find the overall CCR score for a catcher, each pitch is added together. CCR was multiplied by a negative one to show that a positive number indicates the pitching team is gaining runs where a negative number means the team is losing runs due to their framing skills. Inversely, a positive CCR causes the batting team to lose runs and a negative CCR creates additional runs for the hitters. To further the analysis, there are multiple ways to analyze the results.

Because not every team has near the same number of pitches on Trackman, it can be important to look at the mean Contextual Catcher Runs, or Game Score, to compare players. The Game Score for an individual catcher is calculated by:

Game Score = CCR sum / number of takes * 85.37

It approximates the total runs saved or lost in a single game. In the 2019 and 2020 seasons for Division 1, the median number of takes in a game was 85.37, hence why the formula is multiplied by 85.37.

Another way to visualize the CCR is by breaking up the area in and around the strike zone into 30 zones. For the first plot, the colors are based on the sum of the CCR by zone.

This catcher gained a few runs for his team on the bottom of the zone and on the outside of the plate (to RHB). He did not do as well on the inside part of the plate by losing runs. It also appears he lost runs on a couple of zones inside the strike zone as those pitches should have been strikes but ended up being called as balls.

For the two zones in the middle of the strike zone, there shouldn’t be too much significance on the score if the number is close to zero. These parts can become easily negative because of one pitch as it has a very high called strike probability and there’s a good possibility it was an umpire error or an issue with technology misreading the pitch location.

To better interpret a catcher’s season performance, it should also be compared to the rest of Division 1 baseball. In the following plot, each zone is colored by the CCR with the text displaying the zone percentile.

By showing the percentiles, it is noticeable that this catcher is well above average on the bottom of the strike zone and on the outside zones, including the top and right-hand sides. He performed below average on zones that are on the inner part of the plate for right-handed hitters.

CatchCS

The other method — which doesn’t take the game situation into account — was based on how a catcher kept strikes as strikes and stole strikes from outside the zone. This metric is calculated by:

For a strike: CatchCS = 1-Called Strike Probability

For a ball: CatchCS = Called Strike Probability * -1

A positive number indicates the catcher is stealing strikes at a greater rate than losing them and a negative means the catcher is losing strikes more often. Visualizing this metric is just like the visuals for CCR. Using the same catcher from 2019, here are his CatchCS values and percentile ranks:

From these results, this catcher steals strikes on the bottom and right sides of the strike zone while losing strikes on the left side (using the catcher’s view). An interesting note is that the two metrics (CCR and CatchCS) are nearly identical for each zone.

Based on his percentiles, he is above average in getting strikes on the bottom and right side of the plate. He does have some work to do on the left side and top right as those percentiles are much lower compared to the nation’s average.

While no catcher will be red in every zone for either, this is still incredibly useful to identify where the individual does well and what areas should be worked on in practice.

An Alternative Approach

Another way to evaluate framing skills is to look at a catcher’s percentage of called strikes in the Edge Zone. We define the Edge Zone as one ball-width inside the strike zone and one ball-width outside the strike zone. Anything above 50% means the catcher is more likely to get a strike in the edge whereas anything under 50% is unlikely to get a called strike.

Baseball Savant explored this approach here. Using Statcast data, they broke up the Shadow Zone (our equivalent to the Edge Zone) to evaluate the receiving skills of MLB catchers.

Limitations

There are a few limitations found with this project. First, a team that allows runners on base and puts itself into hitter counts more frequently will have additional opportunities to add to their CCR total in high-stress situations. This makes CatchCS useful in evaluations as it doesn’t take the game situation into account.

Additionally, a catcher’s ability to frame is not always the cure for gaining or losing strikes. Simply a catcher may not get the call he desires because a pitcher missed his spot or there was a cross-up on the pitch types. Those situations can easily fool an umpire and call a pitch a ball at no fault of the catcher.

Similarly, a team’s pitching philosophy may impact their catchers’ framing values for certain areas of the zone. For example, if a program hangs their hat on being able to pitch inside with consistency, their catchers may have a positive inner-third score that reflects that philosophy. On the other hand, this philosophy can impact their scores in the outer-third of the zone. If a pitcher misses away when the catcher sets up in, it will be difficult for the catcher to maintain positive scores in the outer third. All of this nuance needs to be taken into account when analyzing these numbers, and for any individualized development plan that would be created with these numbers in mind.

Unfortunately, pitch-tracking technology is not available for every college ballpark like it is in professional baseball. This leads to disparities in the data on the number of pitches a catcher received in a season. By looking at the distribution of the number of pitches and the season Game Score, the amount of variability decreases once catchers receive at least 250 pitches resulting in a take. This is approximately three games worth of called strikes or called balls. Because of this, 250 was used as a threshold for catchers to qualify for CCR.

It was comforting to see the teams who are perennial conference championship contenders and/or had catchers drafted on day one of the amateur draft identified as the top performers of these metrics.

Player Development Uses

The catcher framing metrics highlighted in this article can provide a quality starting point when assessing the strengths and weaknesses of a catcher’s ability to receive. With each team’s pitching strategy in mind, their catchers’ heatmaps can be tangibly utilized to ensure their development is focused on the areas of the zone where their pitchers are attacking and having the most success. Quite simply, the goal is to gain strikes in the area of the zone that their pitchers are attacking the most and thus, maximizing their team’s ability to prevent runs. If the catchers meet this goal, the team’s ability to win games is undeniably enhanced. If they’re not meeting this goal, then a program can use the data to reevaluate how their catchers are training, and appropriately deploy an individualized development plan that reflects the desire to prevent runs with the team’s pitching strategy in mind.

Alternatively, a team could try to tailor their pitching strategy to reflect the strengths of the catcher. In the heatmap examples from above, the team could pound the bottom of the zone as that’s where the catcher is stealing strikes and preventing runs at an above average rate. Although with this method, a team may find that it’s more sensible to develop their catchers to maximize their pitcher’s respective strengths, and not vice-versa.

Final Thoughts

As pitch-tracking technology becomes more prevalent across Division 1 baseball, it will be easier to evaluate a catcher’s ability to receive. Also, there will be more ways to look into the concept of catcher framing by taking the park effects and the pitcher into account, but also identify how catchers develop as their collegiate careers progress.

At the end of the day, the lowest hanging fruit for a catcher to maximize their ability to impact the game the most is through receiving. A team can more effectively put together a development plan for its catchers by utilizing the resources presented in this article, in turn achieving the goal of preventing runs and winning games. The pitchers will get more strikes, increase their strikeout rate, decrease their walk rate, etc. With the improvement of these metrics, the team certainly accomplishes that goal.

Note: This article received contributions from Creighton Rudolph.