AI Winter is coming
--
It happened before and it will happen again — very soon. It is not a matter of if, but a matter of when.
The term AI winter is not even a fancy term made up by me to clickbait you into reading this article but is actually a well-known term in the AI industry. The reason for that are the two AI winters we already have experienced in the 20th century.
What even is an AI winter?
An AI winter has nothing to do with snow, nor the seasons. It is simply about the regression of science funding, startup funding and the general interest in AI. The cause for such a regression is the realization that many hopes and promises can not be fulfilled or in other words: AI is massively overhyped.
Sure, AI is appealing and there is a lot to discover but it would be naive to believe that huge scientific leaps forward could be achieved easily in a short period of time. Sometimes even the enormous difficulty of simple problems, like knowledge representation or logical reasoning, are heavily underestimated.
The name Artificial Intelligence might not be that generous for the field, since it leads to high hopes and misconceptions such as AI going to replace many jobs in the next decades. Even though, the main focus of research is not AGI (artificial general intelligence) at the moment but narrow AI instead, many people still have this image of some kind of terminator or super smart robot in their head when they think of AI. This leads to a hype.
There is a strong correlation with Gartner’s hype cycle which tries to explain that a new technology triggers unrealistic expectations at first leading to much research and investigation into this technology. After realizing illusions, the research and the general interest sees a drastic decrease when it first allows for real progress through slow and steady improvements.
It is very similar with AI except, that we did not have the state of the “Slope of Enlightenment” where the real and steady improvements happen.
Past AI Winters
The first AI boom happened in the 1950s which has also been titled as the age of reasoning and prototype AI. But soon in the 1970s a decade-long AI winter had started.
A new AI boom happened in the 1980s when expert systems which rely heavily on knowledge representation came along trying to reproduce human decision-making. But those achievements could not satisfy the expectations because knowledge representation was only applicable to few areas and due to the lack of data and computing power other algorithms were not feasible.
The next AI winter came and stayed for a while until the amount of data and the compute power increased almost exponentially, leading to the possibility to do deep learning. This is where we are right now, and it could easily be that the pattern reoccurs.
Why will it happen again soon?
It is comparable to a bursting bubble. Whenever the value of something is overestimated artificially high, the bubble has a high potential to burst as soon as the general misconceptions shifts towards an informed common knowledge.
Performant AI nowadays, mostly deep neural networks, are considered to be black box models which perform tons of floating point number calculations to predict X, or Y, or etc. … These models can be very accurate because they are universal polynomial function approximators having hundreds and thousands of neurons to abstract the function over time. Whenever you ask yourself how such a model came to a conclusion there are very limited ways to get to a reasoning of the model, because the flow of floating point numbers through numerous neurons is not humanly interpretable. That is why those models do not make steps forward in the research area of reasoning and knowledge representation which is very concerning in the long term because we need models to have understanding instead of just doing “dumb” pattern detection.
E.g. a neural network which detects a dog in an image might be predicting that there is a dog in the image but it has no real understanding of what a dog is and what it does besides its visual patterns.
You might think that a neural network which is able to classify a dog and a cat in an image could easily also classify a bird in an image. But here is the hook, the AI would not be able at all to classify a bird or any other animal without relearning everything it already has learnt with the addition of birds. This leads to the obvious conclusion, that reasoning and knowledge representation are important to make some sort of association possible. Whenever a human thinks of let’s say a dog we may see the word spelled out in our head, as well as see the visual representation, as well as some dog names, etc., …
The reason why associative memory is important because it is needed to have multitask models (which are a first step towards AGI), which are able to do more than just one thing. Instead of just classifying an animal in an image, the AI could also be able to reason about animals in a sentence. It should understand that the word “dog” and the actual image of a dog are strongly related and mean the same thing.
Conclusion
Because almost all the AI research today happens in the area of deep learning trying to make neural networks more and more performant or robust, the big problems are not getting solved. If there is no paradigm shift happening towards reasoning based AI the next AI winter will come soon.
An optimistic perspective
Although there are many odds against AI at the moment, there is still hope. One research area that might combine the effectiveness of deep learning and the reasoning are logical neural networks which try to map neurons to logical predicates consisting of operators like AND, OR, … The promise is that neural networks would no longer be black boxes and a knowledge representation could be transferred more easily to other models.
To be fair, this approach is not so effective as of today because many information gets lost when mapping the neuron states to get logical predicates. Here is where the hope lives to find a solution to exactly such problems and make the dream of AI reality.
Join Medium with my referral link:
If you enjoyed this blog post and want to read unlimited on Medium, I would appreciate if you use my referral link to create a Medium subscription:
