Ted Williams at bat during the Hall of Fame game, 1955

The Paradox of Lawyer Skill and Luck

Baseball has been America’s pastime and one of the more interesting parts of the game is watching a great hitter. On a warm, summer afternoon, you can forget many cares sitting along the first base line watching a power hitter lay into a curve ball just touching the outside corner of the plate. The swing is perfectly timed, the crack of the bat against the ball is unmistakeable, and the gentle arc of the ball as it flies over the right-fielder’s head and then over the wall is an invigorating sight.

Hitting a baseball often is described as one of the hardest things to do in sports. Great hitters make the incredibly difficult seem easy. But why is it so hard?

Start with the ball. It has a center made out of rubber or cork. The center is wrapped in yarn (up to one mile) and then covered “with two strips of white horsehide or cowhide, tightly stitched together.” The ball is approximately 9.00 inches in circumference, slightly less than 3 inches in diameter, and weighs around 5.00 to 5.25 ounces.

The batter is armed with a wooden stick up to 42 inches long that weighs around 33 ounces. At its most substantial girth, it is around 2.75 inches in diameter.

The pitcher throws the ball, which can reach speeds approaching 100 miles per hour. By manipulating the release point, spin on the ball, and other factors, the pitcher can alter the ball’s trajectory. The pitch might be a fast ball, curve, slider, cutter, or changeup. In fact, there are around twelve different pitch categories.

The distance from the pitcher’s mound to home plate is 60 feet 6 inches. The ball will travel that distance in roughly 0.40 seconds. That means in less than half a second, the batter must

  • determine the type of pitch;
  • determine if it’s a strike or a ball;
  • determine the speed of the pitched ball;
  • finish his stride and get his foot down; and
  • get the bat to the ball.

An average batter successfully hits the ball and gets on base a little over one out of every four times at bat.

While practicing law isn’t hitting a baseball, lawyers feel clients give them an equally difficult challenge. Clients want lawyers to provide better services, at lower cost, more quickly. Lawyers say they offer services that meet what clients want, but that clients actually want something else (lawyers are still trying to figure out what “else” means). Lawyers say clients are not explicit about what they want, just that they don’t wan’t what they currently get. Clients say they are explicit, but that lawyers aren’t listening. Clients have become increasingly dissatisfied with the services offered by large law firms. Today, only one-third of in-house lawyers say they would recommend their primary outside law firm to another in-house lawyer. A bit better, but not by much, than the typical batter’s hitting average.

The client-lawyer debate points out that the legal profession is a collection of paradoxes and it is hard to sort them out. But a book by Michael Mauboussin — managing director and head of Global Financial Strategies at Credit Suisse, an adjunct professor of finance at Columbia Business School, and chairman of the Board of Trustees at the Santa Fe Institute — might help with one: the paradox of skill.

In The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing, Mauboussin explains the paradox of skill and why it yields a surprising result: the more skillful we get the more luck plays a role in the outcome of what we do. To help us get through this paradox, Mauboussin relies on America’s favorite pastime, baseball.

Ted Williams, Baseball Digest, May 1949

The Splendid Splinter

While everything about baseball is debated, many aficionados agree that Ted Williams, known as The Splendid Splinter, was a fantastic batter — in fact, the greatest ever.

Williams compiled legendary career numbers: a .344 batting average, 521 home runs, and a .482 on-base percentage, still the highest ever. His batting average is the highest of any Major League Baseball player with at least 302 home runs.

For the major league baseball season ending in 1941, Williams’ batting average was .406. No other professional baseball player since has exceeded .400 for a season. To put this in perspective, the average batter got a hit 27 times out of every 100 at-bats, but Williams got a hit 40 times.

Because hitting a baseball involves the pitcher and the batter, we can’t define the performance of a batter based on his skill alone. From the batter’s perspective, how well he hits involves a combination of skill and luck. In this case, “luck” means a large set of variables, such as what combination of pitches are thrown, the wind at the particular moment each pitch crosses in front of the batter, humidity, placement of the sun, and so on.

Stephen Jay Gould

Williams’ performance raises many questions, particularly because no one has duplicated it afterwards. Was he lucky or particularly skilled? To answer that question, Stephen Jay Gould, the late Alexander Agassiz Professor of Zoology and Professor of Geology, Biology, and the History of Science at Harvard University and an avid baseball fan, did what all die-hard baseball fans do — he checked the statistics.

Statistics were not just a mathematical tool for Gould. When we was barely into his 40s he was diagnosed with a deadly form of cancer. The median survival period was eight months, but Gould went on to live another 20 years before a second, unrelated form of cancer took him. After his first cancer bout, Gould became very interested in statistics and what they did and did not say.

Gould studied Williams’ batting performance and discovered something. If you compared Williams’ 1941 average of .406 to a batting average of .380 in 2011, there was no statistically significant difference. In other words, the lower batting average (which some players did achieve) was the statistical equal of Williams’ higher average. Gould wondered what had happened during the 70 years between Williams’ spectacular season and the present.

To start, we should look at the batting score average for all batters. During that 70-year period, it remained relatively constant at .260-.270. At first that may seem counter-intuitive. Batters improved during that period. They had better training and improved in strength and overall athletic skill (ignoring those who improved through chemistry). Professional baseball also became more diverse as players came from many countries. In other words, average major league baseball player quality was improving as players were drawn from a wider and deeper pool.

But, during this same period, pitchers were improving. They also had better training, improved overall skill, and were drawn from that same more diverse population. Better training and a more diverse pool are important to understanding that 70-year period, but they don’t tell the whole story.

Understanding the Skill Paradox

Gould hypothesized and the statistics confirmed that the variation among batters decreased during those 70 years. Statisticians would say the mean remained roughly the same at .270, but the standard deviation decreased. Standard deviation is the measure of dispersion of values in the data set. The higher the standard deviation, the greater the dispersion. A picture helps explain the measure.

The blue line shows a bell curve with a standard normal distribution. The percentage of batters to the left and the right of the mean is the same, and the distributions to the left and right of the median are the same, with gently sloping curves.

The orange line shows another standard normal distribution. But in this distribution, the standard deviation is smaller. The lines to the left and right of the mean are squished closer to the mean. In other words, batters are less likely to fall at the extremes and more likely to fall towards the middle. This narrowing of the bell curve (reduction of standard deviation) comes from consistency in batter skill and, possibly, consistency in pitching skill.

Now things get interesting. If batting is a combination of skill and luck, and skill becomes less variable, then luck plays a greater role in the outcome of a player’s trips to the plate. Luck isn’t changing over time, so the distribution of luck remains constant. At the same time, the distribution of skill is getting narrower (the standard deviation is decreasing). This is the paradox Gould discovered: as skill increases, the role of luck in the outcome increases.

The Lawyer Skill Paradox

Now think about the paradox of skill and lawyers. We can hypothesize that during the years when batters and pitchers were improving, lawyers also were improving. Unfortunately, we do not have any objective data to support this hypothesis. We can look at some recent proxies, such as bar passage rates, and infer from those numbers that lawyers have improved or regressed.

In fact, many people use exactly that approach. The next graph shows the bar passage rates for the past 10 years:

National Conference of Bar Examiners, 2015 Bar Examination Statistics

The drop in bar passage rate may suggest that graduating law students, on average, have less skill than in the past. We need more data to get comfortable with this conclusion, such as whether the bar exam has anything to do with lawyer skill and whether skill after graduation follows a predictable change rate (e.g. increases at a standard rate). We will assume that there is a relationship and that graduates who pass the bar are more skilled.

If the paradox of skill holds true for lawyers as it has in other areas, then as lawyer skill decreases and standard deviation increases luck plays a smaller role in the outcome of a legal matter. The odds that a low-skill lawyer will be pitted against a high-skill lawyer increase and skill will be the determining factor. As batter skill increased, the standard deviation decreased and the bell curve narrowed. Assuming that luck remains constant, luck played a bigger role in an outcome as batter skill increased. As lawyer skill increases, then luck should play a larger role in the outcome of legal matters.

The Paradox for Clients

What does this mean for clients? To explore that question, I will take us through a small thought experiment. Imagine two general counsel. One represents the plaintiff (Paul) and the other represents the defendant (David). Paul’s general counsel could hire high-skilled or low-skilled lawyers, the same as David’s general counsel. Both general counsel believe the lawsuit is high risk. Because of the high risk, both general counsel assume the opposing general counsel will hire high-skill lawyers.

What do we mean when we say “skill”? Lawyers and the legal profession do not have definitions for skill. Clients define skill by using a “I know it when I pay for it” standard. Measuring the skill of a lawyer or law firm is difficult. The services offered vary widely from lawyer to lawyer and firm to firm. Evaluating some of those skills, such as giving oral advice, writing legal briefs, or concocting novel arguments, involves many judgments. Without objective and quantified measures, clients and lawyers themselves have measured quality by using what behavioral psychologists call “substitution.”

We all use the substitution heuristic. When we have difficulty evaluating something, we look for easier things to evaluate and substitute those evaluations for measuring the more difficult thing. It is very difficult to evaluate the economic policies of candidates for president, but we can evaluate whether we have confidence in what the candidates say. We substitute our confidence ranking for ranking the policies themselves.

When it comes to lawyers, we use law school reputation instead of evaluating actual skills. Ask a client whether a Harvard Law School graduate is more or less skillful than a night school graduate of William Mitchell College of Law and clients will pick the Harvard graduate. Yet, Warren Burger (a night school graduate of William Mitchell College of Law) became Chief Justice of the Supreme Court of the United States in 1969 putting him ahead of Harvard Law School graduates. Chief Justice Burger’s skills (some political, some legal, some other) got him the job where other lawyers’ skills did not get them nominated.

If Paul’s general counsel hired a low-skill lawyer, he would assume his client is at a disadvantage. David’s high-skill lawyer would “outlawyer” his low-skill lawyer resulting in a loss. David’s general counsel would make the same assumption. Neither general counsel will take that risk. So, both will go for high-skill outside lawyers. (While it is possible both would go for low-skill lawyers, in the real world such a low-low outcome is highly unlikely.)

As soon as both general counsel choose their high-skill lawyers, they have increased the role luck will play in the outcome. Two, high-skill lawyers bring equivalent analytical skills, brief writing skills, and case management skills to the lawsuit. They effectively nullify each other leaving the outcome to other variables, which we are calling luck.

In economic terms, the two general counsels have paid a high price (assuming it costs more to hire high-skill lawyers) to increase the odds that their lawsuit outcome will be based on luck. Another way of looking at the situation is to describe it as a commodification of services.

Richard Susskind uses commodification in Tomorrow’s Lawyers to mean legal services so “commonplace and routinizable that [they] can be made available, in open-source spirit, on the Web.” I am talking about a different form of commodification. In our thought experiment, the services are bespoke, but because of the equally matched skill levels, the advantages they offer are nullified. The pitcher and batter have met their equals. The outcome of the battle depends on luck. The services, though high-end specialized, do not provide an edge to either party and so, while bespoke, are expensive commodities.

Finding the Client Strategy

The best strategy for a client in a market with declining lawyer skill is to get picky when choosing a lawyer. In a market with a high level of variability (a high standard deviation indicating a broad bell curve), picking a skilled lawyer means the client, on average, is leaving less to luck.

Here, the paradox comes in again. When skill levels are high, general counsel should use factors in addition to skill to choose their lawyers. Luck plays a significant role in the outcome, so differentiating factors other than skill become important in the lawyer-client relationship. This idea lies behind many of the battles for clients that exist today. It is hard for many large law firms to distinguish themselves on skill alone, especially for mainstream legal services. But, they do not have other factors to rely on to distinguish themselves. That leaves price, and competing on price is a race to the bottom as well as a signal that your firms’ services are undifferentiated.

We have considered skill, but we also should consider risk. Risk can be defined many ways, but corporations frequently use three categories to evaluate all types of risks: financial, reputation, and operational. A general counsel considering which lawyer to retain in a lawsuit will consider financial risk to her client, whether the lawsuit has a significant risk of damaging her client’s reputation, and whether the lawsuit could affect her client’s operations. The greater the overall risk, the more likely that general counsel will focus on maximizing skill brought to solving the matter.

Because clients substitute reputation for objective skill evaluation, they gravitate towards 1) large firms, 2) with the “best” reputations (from sources such as surveys, industry gossip, and colleagues), and 3) who typically cost the most (price is another proxy for quality). According to The American Lawyer, over time we have seen a separation among firms ranked in the top 100 firms by revenue. A cohort of mostly New York-based firms have moved into the top performing firms (consistent growth in revenue and profits). These two-dozen or so firms seem to thrive by getting the priciest, riskiest, reputation-risking work. The remainder of the AmLaw 100 earn money by handling the less-risky work (before everyone yells, this is a rule with many exceptions).

If we assume the paradox of skill holds true in the legal profession, we have an interesting result. As clients assign their riskiest matters to the most skilled attorneys, we have legal battles between skilled opponents. The law firms for each of the clients are presumably somewhat evenly matched in skill. Since skill has increased and both clients have skilled lawyers, the paradox of skill tells us that luck should play a greater role in the outcome of legal matters. That is, when clients on each side of a lawsuit hire skilled outside lawyers, they “elect” to roll the dice more on the outcome than when only one client hires a skilled outside lawyer or both hire less-skilled lawyers.

The Paradox of Not Knowing

We don’t know whether the paradox of skill holds true for legal services. It is another in a long list of questions that deserve study. We talk quite a bit about disruption in the legal industry, but to really assess what impact these changes have we need to treat the legal industry as something worthy of serious attention. We need to collect data and study the links between various inputs and outputs. To date, most of the stories about change in the legal industry are just that, stories. The real paradox may not be one of skill, but why lawyers who claim to base what they do on reasoning and facts are basing the future of their industry on luck.