Musings From Wrigley Field on Legal Metrics
As a Cubs fan, the only real baseball statistic I know stone cold is 108. (For those who share my affliction, particularly those of you who enjoy a good wallowing, here is a depressing list.)
My relative naiveté about baseball statistics notwithstanding, anyone even moderately interested in the game knows that baseball has counted a lot of things since its earliest days of existence. Hits, home runs, errors, stolen bases — these are examples of “counting statistics.” We compare player performance across generations based on the accumulation of statistics. For die-hard fans of the game, the devotion to statistics is a part of what makes the game special.
For those actually making a living in the baseball business, however, the game is about one thing: winning. Player, coach, scout, GM — all are trying to achieve a clearly defined outcome. Within that context, the statistics serve a related yet distinct purpose, one that is a bit more focused and pragmatic than settling bets and arguments at the local sports bar. For those in the business, statistics serve to measure a player’s contribution toward that outcome and to inform immediate decisions that increase the likelihood of winning more games than the competing team.
For most of baseball history, those statistics — the things baseball counts and keeps track of — were viewed through the expert eyes of scouts, player personnel experts, and coaches to interpret and apply their wisdom to the numbers in selecting players and crafting lineups. And for most of baseball history, the statistics available to this select group of shot-callers were exactly the same statistics used to settle arguments at the local sports bar.
Baseball has come a long way from those days, however. Witness the Moneyball phenomena and the increasingly influential role of Sabermetrics in the business of baseball. In fact, the sophistication of measurements — which continues to develop — is quite amazing. While hits and runs are both time-honored mainstays of counting statistics, baseball now has far more granular information like quickness or speed on the base paths.
Wait, what does this have to do with change in the legal industry? Well, there is a lot of discussion about measuring things in the legal industry. Terms like “metrics,” “key performance indicators,” and the like are tossed around willy-nilly. It is true that baseball statistics are hardly simple or free from debate. It is also true that creating objective evaluation of law firm performance is far more complex (if only because the desired outcome is far less clear). And, yes, as I wrote in a prior essay, we still are not very good at simply counting things.
Still, I believe that the movement exemplified by baseball — the maturation from counting to measuring to evaluating impact on outcomes — provides lessons from which we can learn and apply to the legal industry.
Value measurements — weather in baseball or in law — require a point of view
Take, for example, WAR. No, I do not mean the Edwin Starr song (go ahead, good luck trying to get that song out of your head). Wins above replacement (“WAR”) is an effort by the baseball community to express the array of statistics around a player’s input into the game through a single metric demonstrating the player’s impact on the outcome of the game. I am not about to try to explain the long list of assumptions or the math behind WAR. Nevertheless, I found a few things quite interesting (and in one way or another, these ruminations got me thinking about the current state of measurement in the legal industry).
1. WAR is a composite statistic that combines a number of metrics. This represents an (imperfect yet admirable) effort to quantify the holistic contribution of a given player to the wins racked up by the team. At least in theory, WAR should enable teams to benchmark the contribution of each player, albeit against that of a “mythical “replacement” player who can be acquired on short notice.” (Clearly, WAR has its share of dissidents. More on this below.)
2. A sidebar on detractors. I suspect that there are a wide range of attitudes toward the appropriate usage of WAR in evaluating players and the overall utility of advanced statistical analysis in setting roster strategy. (It’s true that the extent of my knowledge on this topic comes primarily from watching Brad Pitt in the movie, but I rely heavily on my lessons of human nature when I say I would be prepared to bet that the issue probably attracts the full spectrum from devotees to skeptics.)
Speaking as one who has lived among skeptics for many decades, it’s certainly true that statistics can only provide a guide. Not all things are quantifiable and the things that are certainly do not guarantee an outcome — particularly in areas of human endeavor.
Nevertheless, I can see how advanced composite measurements like WAR could add structure and prioritization to ball clubs’ decision-making by grouping roughly similar players by marginal utility — in essence, they attempt to isolate how much one variable (a given player, in this instance) will affect the desired outcome (wins), more or less than any other variable (a player within the same tier, a player in another tier, or the handy replacement-level player). At the very least, I am sure the statistic makes for ever-more advanced depth charts and tier lists for the fantasy baseball aficionado.
3. There are apparently several different variations of WAR, including adjustments for offense, defense, position played; even accounting for those variations, the calculation also depends upon which organization is doing the math.
4. The variations in methodology notwithstanding, all of the input statistics that feed into WAR are completely public. That means all of professional baseball is still working off a shared set of master data. In other words, ball clubs more or less have parity in their access to data — for each baseball franchise, differentiation comes from their approach to using that data and in the analyses on which each organization bases its decision-making.
5. In order to develop a winning Sabermetrics methodology — in essence, to effectively apply advanced statistical measures to decision-making — someone within each ball club actually needs to understand the game, its mechanics, win conditions, and the variables at play, all as the necessary backdrop context for the implications of each statistic. Further, each ball club must also formulate a winning strategy that is unique to the organization. This strategic thinking frames each ball club’s unique approach to winning, while statistical analysis merely provides the fodder — though with great quantitative rigor — to translate that approach into actionable tactics and decisions. It is the specificity and tailoring of each analysis that accounts for each ball club’s unique assets and constraints: capital resources, existing contractual commitments, and the like.
So we ask again: what does this have to do with the legal industry? Plenty.
Lesson #1: Skeptics will remain skeptical, i.e., “haters gonna hate”
Despite the apparent maturity and sophistication of Sabermetrics, a diverse subset of sports commentators decry the risk of it “ruining” baseball. I cannot help but be reminded of the impassioned and persistent resistance of some (not all) lawyers to the notion that the magical and unique work they do could ever be reduced to a set of numbers. There is a school of thought in the legal profession that as metrics and analytics advance in rigor and utility, these advances would hit a critical mass, a decisive tipping point at which the skeptics, one and all, are won over. Well, one can hope.
Lesson #2: Ball clubs must define how they plan to win, but legal organizations must define what winning means to them
In his foreword to the Blickstein Group’s 2015 Law Department Operations Survey, David Cambria made the observation with respect to why the industry is so slow to move to different pricing arrangements: we “revert back to the old, steadfast ways. We decide it’s too hard to figure out, it’s too risky, it isn’t the right matter or it’s not clear what qualifies as a win.” (emphasis added).
Unless we are willing to work to define the desired outcome, based on the realistic needs of the business, it becomes virtually impossible to get off “Go,” to advance beyond simply counting things, often for the sake of saying we are counting things.
Each company must define for itself what its particular needs, priorities, and constraints are before it can establish meaningful parameters to measure the performance of any given lawyer, service provider, or service model. This requires a lot of legwork and brainwork, and more importantly, it requires commitment over a period of time — it is not something that happens overnight. Defining the results — the outcome — and communicating those outcomes require difficult conversations. It requires real, candid, and sustained dialogue between the end user and the legal department and then between the legal department and their outside providers.
Lesson #3: The collaborative build of public/shared data pools probably deserves more attention than the sustained search for The One Metric
An interesting contrast across baseball and law is that baseball statistics (like most sports stats) reside in the public domain, while most troves of legal data reside in jealously guarded, privately owned, or just plain scattered-across-outdated-and-inaccessible silos. This creates interesting implications for the rapidly growing coalition of data disciples and analytics evangelists in the legal industry.
Some, like Ron Dolin, advocate for a bottom-up approach to standardized legal metrics in targeted areas of practice, while others are pushing for broad modernization of government data and improved governance of docket information for public consumption. Still others are designing new businesses to monetize new capabilities in the aggregation and analytical application of such data stores.
In many circles across BigLaw and Fortune 500 in-house departments, there is an ongoing and active gray market of operations folks and quantitatively-minded lawyers trading white papers and spreadsheets touting the value of specific metrics (and often the ease with which it can be operationalized). While I admire the energy and commitment of many I see in this community, I often wonder whether we are all looking outward for the key to unlock the mysteries of metrics — when we might actually be better served by looking inward. Until we can operate from a broader base of generally available data, the search for a common metric will be elusive.
The fact that there are significant challenges to gathering data do not detract from the importance of the task. We need to develop a common language and common set of data to allow us to develop the type of metrics that will drive change.
Nevertheless, it is true that the best data (and by best, I mean most likely to generate meaningful insights for more rigorous decision-making) will generally be specific to the situation.
This does not mean that each and every organization must develop entirely bespoke systems and infrastructure to support measurement. It does mean, however, that the most incisive measurements, particularly on quality and overall effectiveness of legal work, will continue to be tailored toward (a) business objectives and legal outcomes that are specific to the client’s situation and (b) mechanics that are designed for the actual data structures and workflow processes in play.
Just like in baseball, the metrics need to be applied within the specific circumstances and strategy of the organization.
Come back May 11 for Steve’s next essay — “A Tale of Two Metrics: Diversity and Service Effectiveness in Legal”