🦉 10x curiosity — Issue #67 — Sorting luck from skill and reversion to the mean
Are you fully acknowledging the role luck is playing in the outcomes? Does a good outcome reflect your good skill, or did you just get lucky?
If you haven’t already then sign up for a weekly dose just like this.
Thinking…
So you aced the test or got a terrific outcome in that project at work. Pat yourself on the back. You put in the hard hard yards and have been justly rewarded. Maybe you are unhappy with someones performance (a team mate, kid or colleague) and you let them know they have to lift their standards. Thanks to you and that motivating intervention, the outcome next time around is better. Lucky you stepped in or who knows how bad things could have got.
In these examples are you fully acknowledging the role luck is playing in the outcomes? Do you understand the inherent variability that can come from the same process with changing inputs? The outcome of any given event is made up of skill + luck, and depending on the event that outcome can be disproportionately weighted toward either extreme. Poor outcomes can reflect the combination of some skill and a lot of bad luck. Does the fact it is different next time around reflect a change in your process or just a random change in your luck?
Daniel Kahneman highlights that any system that combines skill and luck will revert to the mean over time. When he was asked to offer a formula for the twenty-first century he submitted:
Michael Mauboussin writes in his book “Think Twice — Harnessing the power of Counter intuition” :
Say results are part persistent skill and part transitory luck. Extreme results in any given period, reflecting really good or bad luck, will tend to be less extreme either before or after that period as the contribution of luck is less significant.
This concept is important in predicting the future as
an outcome that is not average will be followed by an outcome that has an expected value closer to the average. While most people recognize the idea of reversion to the mean, they often ignore or misunderstand the concept, leading to a slew of mistakes in their analysis.
Consider the under sampling of failure highlighted in this HBR article by Mauboussin -
The most common method for teaching business management is to find successful businesses, identify their common practices, and recommend that managers imitate them.
If causality were clear, this approach would work. The trouble is that the performance of a company almost always depends on both skill and luck, which means that a given strategy will succeed only part of the time. Some companies using the strategy will succeed; others will fail. So attributing a firm’s success to a specific strategy may be wrong if you sample only the winners. The more important question is, How many of the companies that tried the strategy actually succeeded?
The lesson is clear: When luck plays a part in determining the consequences of your actions — as is often the case in business — you don’t want to study success to identify good strategy but rather study strategy to see whether it consistently led to success.
How do you avoid mistakes associated with reversion to the mean? Here’s a checklist from Mauboussin (Think Twice) that may help you identify important issues:
1. Evaluate the mix of skill and luck in the system that you are analyzing.
- A simple test of whether an activity involves skill: ask if you can lose on purpose… if you can lose on purpose, then skill is involved.
- When something good happens, we tend to think it’s because of skill. When something bad happens, we write it off to chance. So forget about the outcome and concentrate instead on the process.
2. Carefully consider the sample size.
- Daniel Kahneman and Amos Tversky established that people extrapolate unfounded conclusions from small sample sizes.
- The more that luck contributes to the outcomes you observe, the larger the sample you will need to distinguish between skill and luck. Baseball is a good example. Over a 162-game season, chances are good the best teams will rise to the surface. In the short term, however, almost anything can happen. … “In a five-game series, the worst team in baseball will beat the best about 15 percent of the time.” You do not see this in chess or tennis matches, games in which the best player almost always beats the worst, regardless of time frame.
- When a large number of people participate in an activity that is influenced by chance, some of them will succeed by sheer luck. So you have to scrutinize even long, successful track records in fields with lots of participants. Investment track records are a good example.
3. Watch for change within the system or of the system.
- Not all systems remain stable over time, so it’s important to consider how and why the system has changed. One obvious example is individual changes in skill level. An athlete’s age is a good example. In many professional sports, athletic skill improves through the late twenties, at which point it begins to steadily deteriorate. So above-average athletes revert to the mean over time as a consequence of diminished skill.
- Further, the system itself may change. Stephen Jay Gould analyzed why baseball has not seen a player sustain a .400 batting average for a complete season since Ted Williams in 1941. …Gould showed that while the mean batting average in the major leagues has been fairly stable over the years, the standard deviation has shrunk from roughly 32 percent in 1941 to about 27 percent today. The bell of the bell-shaped distribution has a narrower width than it used to.
4. Watch out for the halo effect.
- Much of our thinking about company performance is shaped by the halo effect … when a company is growing and profitable, we tend to infer that it has a brilliant strategy, a visionary CEO, motivated people, and a vibrant culture.
Once you’ve embraced the paradox of skill, you’ll see that it’s appropriate to have an attitude of equanimity toward luck. If you’ve done everything you can to put yourself in a position to succeed, you should accept whatever results appear. Some days you’ll be lucky, and the results will exceed your expectations. Some days the results will be disappointing because of bad luck. The best plan will be to pick yourself up, dust yourself off, and get ready to do it again tomorrow
From his book “The Success Equation- Untangling Skill and Luck in Business, Sports, and Investing” — Mauboussin suggest a final important point that the very effort that leads to luck is a skill:
Say that you need to complete ten interviews with prospective employers to receive one job offer. Individuals who seek only five interviews may not get an offer, but those who go through all ten interviews will have an offer in hand by the end of the process. Getting an offer isn’t luck, it’s a matter of effort. Patience, persistence, and resilience are all elements of skill.
Skill is best defined as a process of making decisions. So here’s the distinction between activities in which luck plays a small role and activities in which luck plays a large role: when luck has little influence, a good process will always have a good outcome. When a measure of luck is involved, a good process will have a good outcome but only over time. When skill exerts the greater influence, cause and effect are intimately connected. When luck exerts the greater influence, cause and effect are only loosely linked in the short run.
________________________________
Let me know what you think? I’d love your feedback. If you haven’t already then sign up for a weekly dose just like this.
Links that made me think…
The Long Way Round: The Plane that Accidentally Circumnavigated the World — medium.com
The morning of 6th January 1942 was going to be a cold one. Not that this was unusual for New York, mused the night-shift air controller at LaGuardia’s tower, but it did mean he’d have to wrap up…
Leapfrogging Tech Is Changing Millions of Lives. Here’s How — singularityhub.com
In developing countries, particularly African ones, millions of people are skipping the technological evolution process, leapfrogging over now-obsolete technologies and going straight to modern fixes. As a bonus, these often happen to be green, sustainable, and relatively cheap.
How I failed my daughter and a simple path to wealth — jlcollinsnh.com
An interesting read on a simple plan to grow your wealth.
A Mental Model to Leverage Information Overload as Creative Fuel for Problem Solving
Surf the internet through the lens of your most important questions. These questions will organize your attention.
You will discover your own answers to your most important question. And new insights will emerge.
The problem is not information overload, but that you’re not putting up a useful lens to filters to give you the information that you need.
Once you have these filters setup, you can generate novel insights
