The Future of the MLB R&D Department
WHAT’S NEXT?
It’s a question I think a lot about. I’m a forward-thinking individual who has spent most of his life focused on future possibilities rather than the past or present. It’s what allowed me to relentlessly pursue a career in the competitive field of baseball analytics and succeed, contributing to the R&D efforts of three MLB front offices and having close contacts in at least a dozen more.
What drew me to this industry as a high schooler was the opportunity to make an impact by doing whatever it takes to find and exploit sustained competitive advantages. A job that requires creative, out-of-the-box problem solving in a competitive environment where good work means more fans going home happy? Sign me up.
That was my vision. But after years of networking with people in these departments and eventually becoming an analyst myself, I’ve realized something important. My experiences in this field were largely unrelated to finding a “Next Big Thing.”
Instead, the prevailing mindset in these departments appeared to be one of low-risk marginal improvement. That is, taking a process that already exists, making it a bit better, and repeating endlessly. Do this enough times, and those small contributions start to add up, the thought goes. The role of the modern R&D analyst, in my experience, is more or less about facilitating this cycle than it is about creatively attacking new problems.
But is this how R&D departments will be operating 5, 10, or 20 years from now?
For a person whose life revolves around the question “what’s next?”, spending time sprucing up an advanced scouting report or tweaking a pro projection system proved ultimately unfulfilling (and contributed to my decision to discontinue working in these roles full time). Though I recognized the value of those things, my brain couldn’t ignore the opportunity cost. With the time and resources organizations spend making marginal improvements to their core systems, couldn’t they be getting more ROI elsewhere? The question has gnawed at me since I began seeing how these departments really operate.
The main problem solving tool currently used by club analysts is predictive modeling, a technique that always reminds me of this image I first saw in an elementary school math class:
As I have become very familiar with this kind of modeling, this image shows how I like to think about predictive models at their simplest. The analyst puts in some information (the data), asks the computer to make a “rule” (the model), and then the computer outputs some information to the analyst (the final prediction).
This is an oversimplification of the complex process (sometimes called machine learning) that is the cornerstone of most R&D departments today. Put the data in, make the model, get the prediction out. There is an interesting wrinkle during this process, though. Analysts usually have to make an important decision and choose between getting good output from an interpretable model or getting even better output from an uninterpretable “black box” model.
Generally, due to the complexities of predictive modeling methods, analysts can’t have both the best predictions and the clearest understanding of how the model works. Would you trade predictive performance for better model transparency?
In my experience, interpretability is often sacrificed in order to improve model accuracy. Translation: Analysts don’t always know what’s going on inside that model, but it sure does make some good predictions!
I am intimately familiar with the powerful utility of predictive modeling, but what is rarely discussed (if ever, in my experience) are the downsides of relying too heavily on this one analytical technique for all of our baseball problem solving needs.
Often in my experience, it has felt like predictive modeling has been the only tool in the modern baseball researcher’s toolbox. In any situation, there are downsides to having a monolithic set of problem solving tools. Most obviously, there are important problems that are not easily solved by predictive modeling. Limiting yourself exclusively to addressing problems that 1) exclusively require the data that you already have and 2) fit into the framework of a predictive model can needlessly leave significant potential for competitive advantages on the cutting room floor.
Most R&D groups have their core models: ones that predict every MLB, MiLB, and amateur player’s future output (Player Performance Projection Systems), ones that evaluate the quality of each pitcher’s pitches (Pitch Quality Model), ones that predict the outcome of any given batter/pitcher matchup (Matchup Model), ones that tell fielders the optimal place to stand (Shifting Model), and more. In short: teams are focusing on models that answer the core questions of baseball that people have been trying to answer (often with only their instincts and anecdotes) for a long, long time.
Though I cannot prove this, it is my belief that the core models of most teams are somewhat similar in quality at this point in time. Of course some are better than others, but how large is the gap? Each team is using similar modeling procedures and similar datasets to answer similar questions, so there are a limited number of ways they could be building these core models. There is likely plenty of overlap in model results from team-to-team.
Also, each model can only predict the future with a certain degree of accuracy in the best case scenario. So, since teams have been working on models that are aiming to answer similar questions using similar techniques and data as most other teams for long enough to be nearing the upper bound of possible model performance, I have to wonder if these core models are 1) about as good as they are ever going to get with currently available data and 2) similar enough to other teams’ models that their insights are no longer competitive advantages.
For example, You could have the best matchup model in the world but if your opponent has the same one, the model isn’t really helping either one of you in the long run. That is the nature of a zero-sum game like baseball.
It is my understanding that many teams have gone beyond these core models, building additional models that address more specific, granular questions and whose outputs deliver some marginal value as well. But the question remains: are predictive models alone still a team’s best bet at exploiting a sustained competitive advantage in the year 2022? Or is it time to add some tools to the researcher’s toolbox and begin hammering away at understudied opportunities for advantages in addition to existing research practices?
Ultimately, my dissatisfaction with the current status quo in baseball research comes from my inability to see where this is all going. What will club analytics departments be working on in the future to gain a competitive advantage? What is next?
My experience with club R&D thinking patterns makes me fear that the answer is something like this:
“We’ll maintain and tweak our core models until new data becomes available, then integrate it into those models. Meanwhile, we’ll create models to address [relatively low impact] research projects from our backlog.”
But is that the best you can do as professional baseball researchers? Are you really doing all you can to help your team build a long-term advantage? These are questions that tore me up in my time in R&D departments.
To illustrate this question, I present you with a choice. Which of the following options would you choose?
A) For one week, I give you a guaranteed $5 every day
OR
B) For one week, I give you a guaranteed $1 every day *plus* a 25% chance that I give you an additional $150 at the end of the week
I would not fault anyone for choosing option A. At the end of the week, option A is going to net you the most money ($35 vs $5) 75% of the time!
But what if I gave you this option every week for an entire year? Would your answer change?
Over the long haul, the risk associated with option B begins to even out. Getting the big payoff even 25% of the weeks nets you more money than option A at the end of the year.
In this example, option A represents an R&D department focusing only on projects with a small but guaranteed return. Option B represents a department allocating resources to a variety of projects from low-risk to high-risk and from low-upside to high-upside. If only a fraction of the “high-risk, high-reward” projects succeed, their impact could easily be large enough to make the whole thing worth it in the end.
My assertion here is that if teams choose to pursue more high-risk, high-reward R&D initiatives, they are more likely to generate competitive advantages in the long term than teams operating under the current status quo by favoring lower-risk projects. Having leadership onboard with the idea of devoting some resources to projects that might fail would allow for greater upside of R&D contributions.
For the team that wants to accept some risk in exchange for higher upside, let’s evaluate some of the “high-risk, high reward” initiatives that teams could allocate more resources towards.
Evaluating New Data
With new data comes new opportunities. Teams understand this, which is why they have a reputation for investing in all of the third-party data they can get their hands on. In addition to Statcast (provided to every club by MLB), teams use data from a wide variety of outside vendors to help quantify what is happening on the field. The first step in solving a problem in R&D is having the necessary data to analyze, and that’s where these companies bring value.
It is my understanding that teams do spend some resources vetting the quality of these products and thinking about how to integrate them into existing processes, but this task is not a large part of where these departments are allocating their time and energy.
Historically, I’ve observed, innovation in MLB R&D has coincided with the availability of new third-party tools coming to market that help quantify some previously unquantified aspect of the game. For example, teams began shifting when player positioning data became available to them and began helping pitchers alter their arsenals after the popularization of Edgertronic cameras and Rapsodo units.
But there is an issue with relying on third-parties for new information: they don’t capture everything a team might want to quantify. Though it may seem like every single movement on a baseball field is captured and available for analysis, there is a shocking lack of data on several extremely important parts of the game (a topic I’ve explored more deeply here). It is my opinion that some of the most important things we could possibly be quantifying are going completely unobserved by analysts due to a lack of adequate data.
The value of new information is undeniable, so why are more R&D resources not being allocated towards collecting data themselves? Data collected internally would be proprietary by definition and available to only one team (unlike almost all data I’ve ever seen these departments use). Any insights acquired from this data would be an instant competitive advantage.
With so much effort going towards making better predictions with incomplete information, why not try to take steps towards making your available information more complete and improving your analyses that way?
Note: I’m sure some teams are currently collecting proprietary data, but most are likely not doing so at the scale that I think they could/should be to maximize their competitive advantage (which, again, should be the whole point of any R&D efforts in this space).
This leads directly into my second idea for additional/alternative uses of R&D resources to create value.
Studying Baseball at a Fundamental Level
For me, one of the most important parts of problem solving is understanding the problem in as much detail as possible. Unfortunately, my experience is that this is not always prioritized by the modern R&D department. Due to their place within the structure of the front office, R&D groups are typically most focused on delivering products to stakeholders (the GM, AGMs, ML Coaching Staff, MiLB Coaches, other department Directors, etc.) rather than furthering their understanding of the analytical problems they seek to solve.
In addition to working on low-risk projects, some R&D resources could be allocated towards projects intended to improve our understanding of the game at the core level.
You may wonder, aren’t analysts already learning about the game via their current low-risk projects? Oftentimes, the answer is no! Think back to our In/Out function box and how analysts typically chose more accurate models over more interpretable models. The deliverables of many R&D research projects, in my experience, are completed without much meaningful understanding of how the model output was produced “under the hood”. Many times, the machine is learning more than the analyst, and that’s a problem.
Baseball is so complex that we have to develop heuristics (aka shorthands) based on past experience for understanding why things happen the way they do. But these commonly held beliefs about the game can break down when interrogated. An obvious example is pitch sequencing. Little empirical evidence has ever been found that the order of pitches is important in determining the outcome of a plate appearance, but players and coaches swear that the order of pitches is important so the idea still informs pitching strategy at the highest level. It’s a major inconsistency at the heart of our game. The first person/club to reconcile this paradox is likely going to be able to leverage that knowledge as a competitive advantage for a long time.
Let’s examine a different example: the times through the order penalty. We know that a batter’s outcomes tend to improve the more times they face the same pitcher in the same game, but it is not clear why this is the case. Plenty of research has been done trying to determine whether the effect is moreso due to pitcher fatigue or batter familiarity over time. Like many of baseball’s unanswered questions, the times through the order problem is a question of causality, and it is one that cannot be fully answered with pitch tracking data. Well, how do scientists usually address questions of cause and effect? The answer is right under our nose.
Experiments
Running experiments would be an excellent use of R&D resources, in my opinion. In a controlled environment, individual effects can be isolated in a way that they cannot be in the course of a normal game. Collecting this data (which again, would be completely proprietary and inherently advantageous) could help an organization improve their understanding of the underlying processes of the game relatively quickly and cheaply.
In our times through the order example, a simple experiment involving normal baseball games but with random batting orders (where a random hitter from the lineup is selected to take each at-bat with replacement, thus eliminating confounding between pitcher fatigue and batter familiarity with the pitcher) could give us more evidence related to the cause of the times through the order penalty than any of our previous research using observational data. This experiment could be run with games on back fields, in spring training, or in independent leagues, for example.
I am certain I am not the first person to suggest that teams invest in experimentation, so there must be plenty of roadblocks preventing R&D groups from doing so. However, after many conversations on this topic with front office members, I have yet to hear an argument against this idea that was personally convincing. Mentions of player health, using up player training economy, and a myriad of other concerns ring hollow for me since these things could be accounted for in the design and timing of experiments. Collecting this data may be more strenuous than simply having your data delivered directly to you via an API, but the upside here could easily be worth a little elbow grease. Without other persuasive arguments against experimentation, I have to assume a systematic risk-averse mentality or lack of imagination could be playing parts in suppressing this method of scientific inquiry in baseball.
If the organization’s leadership group believes in the potential benefits of experimentation, they should be willing and able to encourage players and coaches to participate for the good of the club’s collective understanding at minimal additional hassle for the participants.
The more I learn about the game of baseball, the more I understand how much I don’t know. I could probably spend the rest of my life studying this game and never run out of questions (and a part of me wants to do just that). This is another aspect of the job that drew me to the field only for me to be disappointed by its absence.
I believe that a team that understands the underlying mechanisms of baseball better than their competition is going to be a team that surpasses its opponents in the long term. The accumulation of this kind of knowledge is not going to happen by accident and it is not inevitable. It would take a conscious investment of resources and time from the entire organization. There is no guarantee that every experiment would deliver value. But in my mind, the potential for a research team to more deeply understand their problems and therefore produce more nuanced solutions is well worth the risk of failure.
Reevaluating Existing Processes
Every team has its processes. How it evaluates players, how it trains players, how it uses players on the field. In my experience, these processes are typically well-established and are far older than the R&D departments themselves which is why it can get a bit uncomfortable when people start asking questions.
For as much time as I have spent breaking down players and researching on-field decisions, I have probably spent even more time thinking about the operations and structure of front offices themselves by asking questions like
“What are we ultimately trying to do here?”
“Why are we doing things the way we are?”
“Is that the best way to accomplish our goals?”
These questions, though well meaning, can be taken as insubordination or criticism of leadership. It is my opinion that these questions are absolutely essential for front office leadership to be considering, but I understand how they may not always be welcome.
Luckily, there are plenty of processes within an MLB organization that are more open to tough questions and new ideas. Take Player Development for example. Helping young players grow their skills until they are Major League ready is one of the more important processes a baseball team has. Every organization now has a similar setup: four minor league affiliate teams where players generally play one baseball game per day for about six months. Batters get about four plate appearances and usually see fewer than 25 live pitches per day.
Why is this the accepted system to develop players?
If our goal is to help our players improve their skills as efficiently as possible, is this really the best we can do? (Of course, the goal of player development has long been intertwined with the goal of creating entertainment and profit in minor league parks around the nation, hence the excess of games.)
A possible innovation in this space could be to research the potential impact of different training methods between (or, gasp, instead of) minor league games. Improvements and innovations in this space seem to have skyrocketed in recent seasons (like those documented in books like The MVP Machine and Swing Kings), many more of which are likely being kept under wraps for good reason. Some teams are currently exploiting this competitive advantage in player development! See, isn’t it fun?
I believe all processes in a baseball organization can and should be questioned (with those least open to criticism likely being the ones most in need of interrogation). I believe the R&D group can be leading the charge of evaluating a club’s internal systems and looking for opportunities for improvement as yet another way to contribute to the welfare of the organization and to the development of sustainable competitive advantages.
The next innovations in baseball are coming. Change and improvement are constantly looming, especially in a competitive field like Major League Baseball. But what will they be? Who will identify, implement, and exploit the advantages behind the next sustained baseball dynasty?
Teams don’t need to wait for the next groundbreaking dataset to show up in their inbox to innovate. Allocating more R&D resources towards higher-risk, higher-reward initiatives is a low-cost way to invest in the future of the team without sacrificing much in the short term. These departments could be collecting their own proprietary data rather than waiting for the next third-party innovation. They could be studying the game at a fundamental level to form more specific research questions and deliver more nuanced research results. Embracing experimentation would be one way to improve access to proprietary data and improve org-wide understanding of the game. Lastly, encouraging R&D members to ask tough questions about internal procedures and systems could yield even more value off the field.
Years of hard work and innovation by club R&D departments have left an obvious impact on the competitive landscape of today’s game and created previously unimaginable value for teams. But as the landscape changes, which clubs will accept the risks of innovating boldly and lead the cutting edge by reimagining the use of R&D resources to have a greater impact and win more games?
