Dialectic of AI: connectionism vs symbolism
The history of AI is a teeter-totter of symbolic (aka computationalism or classicism) versus connectionist approaches.
A brief historical overview
It started from the first (not quite correct) version of neuron naturally as the connectionism. That was a straightforward move, also at that time, it was easier to connect some computational elements by real wires, then to create a simulating model.
However, researchers were brave or/and naive to aim the AGI from the beginning. Even though the development of computers and computer science made modelling of networks of some number of artificial neurons possible, mimicking the mind on the symbolic level gave results much closer to practical problems and the AGI dream at the same time. So, most of the brains and money were directed in this direction.
Unfortunately, with primitive models of reality and the rudimentary ability for learning, the symbolic approach reached its limits despite broad adoption in business and research. It was found out that using even more primitive projections of reality in the models, but adding the ability of training instead of hardcoding and adding rules, it’s possible to get a lot of useful insights and solutions for narrow cases, so the era of machine learning began.
The success of ML was also its curse: each narrow task needs its specific solution, so the zoo of ML models made it a niche at the edge of statistics and computer science.
Then deep learning, which theoretically was there for quite a long time, suddenly became a thing. Actually, a very big thing. Perhaps the most real projects are still based on the traditional ML models, but the best results, the biggest money, and the most attention are on the DL side.
The current state
And here we are at the moment. But something is rotten in the state of the DL art. The time of fast advances has changed to tinkering the settings to get the next 0.1% accuracy and brute-forcing with power consumption which is dangerous for our planet. We are near the limits of what can be done using statistical hacking of reality. The lack in the DL models of common sense, some intuitive physics, and self-supervised continuous learning is obvious even to the leaders of DL mainstream. Not even mentioning that the 20–40 Watt power consumption of the human brain looks like a cruel mockery of the megawatts of DL supercomputers.
Basically, the only plausible solution to this problem which is discussed now is creating a hybrid of DL and symbolic AI with some additional tricks. Nobody is even close, but at least such a Frankenstein monster looks possible (ignoring the power consumption problem).
According to Hegel, the world makes progress by moving from one extreme to another and generally needs three moves to establish the balance. It looks like it’s exactly the case of AI development, where we have had two moves from one extreme to another one: from connectionism to symbolism, and from there to the advanced connectionism. So, the pendulum has to move back one more time, but not to the symbolism as we know it, but something with the best parts of both worlds. Not by just combining them, rather by the exit to a completely new level, through thesis and antithesis to synthesis.
What’s next?
From the dynamics of previous paradigm shifts in AI, we can see some patterns, which can help to guess something about the next shift. Taking to the account generalized measurement of paradigm traction (publications, people, applications, money, public attention, etc) and reflecting on the chart only the difference, you can see the following (it’s just a rough estimate without solid methodology behind it):
The key observations:
- All stages have a similar duration. We can’t be sure about the current one, but at least it doesn’t deviate at the moment.
- All stages start slowly, then have a period of fast growth, and finally, fast decay. Again, we don’t know the part about decay for the current stage yet, but at least the dynamics that we already see looks similar to the previous stages. Also, remember, it’s about the difference, the decay doesn’t necessarily mean a decrease in absolute numbers.
- The scale of every next stage is in times higher compared to the previous one.
We don’t have enough data points to make any solid conclusions from these observations. However, if you think about underlying reasons (hardware and infrastructure development, the inertia of involved people and institutions, the formation of areas of practical application and industries adoption, hype cycle, etc.) they look quite logical.
Will it be different from the next (possibly final) paradigm shift? It’s plausible that there will be some, mostly related to the duration of the slow part of the stage: it has to be much shorter. The main reasons for this are the following:
- Hardware and infrastructure are already good enough to be used without waiting for specialized solutions.
- There is a huge platform for the fast adoption of the next-generation AI created by all existing data-based companies.
- Investors and governments are already educated to recognize this shift as a point of the highest opportunities.
- The technological stack will be much less fragmented, because of the solution universality (for instance, no more separation between computer vision and NLP fields), and a much faster pace of progress.
It’s very difficult to imagine how the transition will be looking, but considering the start of the shift in the near future, it’s safe to say that in ten years the stage will be at its exponential part of the development.
