Are All Complex Systems Always Unpredictable?
Recently I have been reading many popular, social sciences books including books by economist Daniel Kahneman, philosopher Nassim Tabel, sociologist Maclom Gladwell and media writer Derek Thomson.
In their books these authors all discuss complex systems, chaos theory, graph theory and network theory. They all usually arrive at the broad conclusion that all complex systems are unpredictable.
We humans like to overgeneralize, which is one of most common ways we arrive at a wrong conclusion. These writers use some specific observations to make a general statement that all complex systems are unpredictable.
Many complex systems are unpredictable most of the time but some complex systems can be predicted at some point, given the right background and information.
First I want to state that all these writers have interesting observations and thoughts, and present valuable ideas that one can learn from.
My Dad once told me that if I really consider someone intelligent and worth learning from, I have to find something in that person’s views that I disagree with, otherwise I am just blindly following the person’s opinion and not analyzing enough.
In the context of complex systems the above mentioned writers are all prone to overgeneralization. But they arrive at their wrong conclusions in different ways of overgeneralization.
- Kahneman, a statistically minded economist, in his book “Thinking, Fast and Slow” discusses that intuition can work in simple settings with limited outcomes but not in complex systems such as the financial markets. Most of his reasoning is very statistics focused with some shades of psychology. He presents the lack of statistical significance in the performance of professional money managers as a whole and highlights that they do not outperform the index.
However, Kahneman assumes that since there is no statistical significance in the performance of a group of investment managers in public markets that no specific investment manager is able to do this. This is an over-generalization that transfers a conclusion about the total sample to each member of the sample.
- Taleb, a former options trader, in his book “Antifragile” states that complex systems cannot be predicted. He considers complex systems driven by volatility and guided by responses to stressors. He dislikes the simplified mathematical and model based approach to finance and economics that is currently part of the accepted theory. I also dislike and disagree with many parts of the current financial and economic theory.
However, Taleb assumes that because the broad socio-economic system is really complex and highly unpredictable, that other complex systems/subsystems including the economic and financial system have to be unpredictable as well. This is an overgeneralization that transfers the behavior of an entire system to its subsystems.
- Gladwell is a sociologist and in his book “Outliers” discusses the role of chance in drastic outcomes of highly successful people. He argues that people, who we consider outliers, got a lucky break at some point and their disruptive abilities were random and not predetermined. Luck is an important component in life and clearly a shaper of everyone’s future.
However, Gladwell assumes that since luck played such a big role in the destines of some outliers, therefore most people with disruptive impact have succeeded primarily because of luck or unpredictable external circumstances (this is bad logic). He concludes that the complexity of the system does not allow for a predictability of an outlier outcome. This is an overgeneralization that transfers the conclusion from individual instances of outcomes to all instances.
- Thomson, a media writer, in his book “Hit Makers” uses his knowledge and background to illustrate popular phenomena in film, music, graphic art and other forms of media. He traces the random instances that make an obscure media phenomenon become a global media phenomenon. He later discusses some ideas about predictability of hits and chaos theory and extends it to the spread of all new phenomena.
However, Thomson assumes that because popular media phenomena are driven by large degrees of randomness that all disruptive social phenomena are unpredictable. This is an overgeneralization that transfers a certain dynamic in a specific field to phenomena in other fields.
An interesting side-observation to many social science books is that most of them quote the same limited number of social science experiments. They seemingly base their conclusions on the same limited knowledge base.
In a sense most knowledge itself is created by generalization. In social sciences we create knowledge by observing a limited number of outcomes and making a conclusion about a larger set. Therefore knowledge can often lead to overgeneralization.
One contributor to overgeneralization is that most people only have experience in a limited number of settings and fields. Their perspective is shaped by this exposure. Information from a limited number of perspectives makes a person more likely to overgeneralize patterns from one field to fields where one does not have any experience or expertise. We can call this a perspective bias.
The future is really difficult to predict but not all parts of it are unpredictable. Granted this predictability is very difficult at an early stage of a phenomenon but there is usually a point in any disruptive phenomenon, where the phenomenon becomes much more predictable.
However, this tipping point does not occur when the phenomenon’s external expression tips. In reality it occurs much earlier. It occurs at a point when enough related phenomena have created a proportional, positive feedback dynamic to this phenomenon and therefore constitute a strong positive feedback system.
At this point the system is fairly predictable. I specifically said fairly because unpredictable occurrences like a car accident that kills the leader of a movement, a natural catastrophe that devastates an entire city, or a virus outbreak can derail the path of a phenomenon. Therefore diversification in predictions is always useful.
But in absence of such drastic, random influences many systems can become predictable much earlier than visible by their outward behavior.
The key to predicting the future of a complex system is identifying the existence and creation of these proportional, positive feedback loops early enough. One needs to look at a phenomenon as a much bigger system that incorporates everything else that interacts with this phenomenon. And if one is able to identify these proportional, positive feedback loops early, one can predict the future with significant certainty.
The skill of capturing these feedback loops in a field is usually honed by prolonged, in-depth exposure and analysis of similar phenomena in a field. After a while one recognizes common modes of positive feedback in a field and hence becomes better at capturing them early.
A somewhat related thought I want to discuss is the power law distribution that we often hear about in disruption. The reason disruption tends to have a power law distribution is because of this significant positive feedback loop aspect.
A proportional, positive feedback reinforces and strengthens a phenomenon. In the next step this feedback grows, since the phenomenon itself grew. Therefore the feedback reinforces the phenomenon even more and creates a self-reinforcing cycle. One common example of proportional, positive feedback is “network effect”. One can also look at positive feedback in terms of derivative calculus.
Mathematically the power law distribution is simple to explain, as it is the mathematical nature of the exponential function. The exponential function’s derivative is proportional to the function itself. This means that the function’s change is proportional to the current state. Hence as the current state increases by the change, it increases the change of the next state and this cycle starts reinforcing itself.
This type of behavior is very common in nature, as the exponential function is a solution to many partial differential equations that govern the physical world. But in real life there is also a damping function that increases non-linearly with the phenomenon and therefore leads to a finite distribution and not an unstable blowout.
In social systems this damping can have many shapes. Here are some simplified examples:
- For compound interest one mode of damping is the progressive tax.
- For Silicon Valley’s network effect some modes of damping are the high rent prices and talent shortage.
- For a high growth disruptive startup the damping is the inefficiency of scale and bureaucracy of structures.
In the real world there is no pure exponential behavior, because a pure exponential function by its nature would grow infinitely, consume everything become unstable and not exist anymore. Every existing natural phenomenon that is not in current disruption must be in a negative feedback mode within a range around its current equilibrium.
But there are many phenomena that exhibit phases of exponential behavior because of positive feedback loops. For disruption to occur the feedback has to become larger and grow faster than the damping of the system. If one is able to identify this early enough, one can predict some aspects of the future. In a social setting this is difficult and requires expertise and experience, but it is not impossible as many of the above mentioned social science writers claim.
When it comes to the predictability of complex systems, disruption is probably one of the easiest components to capture. Non-disruptive phenomena are usually harder to capture and track.
Since I am a dyslexic, I am prone to spelling and grammar mistakes. Hopefully it does not distract from the substance of the article.
Thank you for reading this article :)