AI is an AI Problem
Not every problem has a thousand solutions, but finding a problem with only one is very rare. That’s why when AI researchers seek to solve problems, they use a method called “optimization” to find not only the solution to a problem, but to find the best solution to that problem.
Imagine the nature of the problem as a mountain. Every time they run an algorithm to seek a solution, they’re looking for the mountain peak.
Finding that peak involves running the algorithm over and over. Like climbers making their way up the mountain, the algorithm tests solutions and then compares them against each other, looking for the best one.
However, if all of the steps start out at the same point, they can only ever climb one mountain. What if it turns out that they’re standing in a mountain range — that there’s a better solution out there, using a different starting point?
To ensure that researchers don’t get stuck on a small mountain range, they use diverse techniques and random restarts while optimizing. This introduces an element of the chaotic, and makes sure that you don’t end up with a poor local optimum. This is a foundational element of machine learning — but it’s one researchers have forgotten when looking at the field of artificial intelligence. Because creating true general artificial intelligence is a problem, and lately, we’re exploring only one path to find its solution.
Researchers have begun to focus exclusively on deep learning, to the potential detriment of optimization. No one knows if deep learning will end up being a local optimum, or if it will be the peak we were searching for all this time.
In the early days of research into artificial intelligence, there were as many pathways to solutions as there were problems to be solved. In fact, the “No Free Lunch” theorem posits that it’s impossible to find a single method to solve all AI problems, and in the past researchers have embraced that philosophy. From Random Forest to Support Vector Machines to Bayesian learning, we have explored countless methodologies.
The problem is that instead of considering each of these methodologies as separate starting points, we often view them as progressions towards a better understanding. In the early 2000s, because of the progress of Support Vector Machines (SVM) and kernel tricks and training problems with neural networks (backpropagation problems), it was incredibly hard to publish a paper on neural networks. Everyone believed SVM was the way of the future, and neutral networks were in the past.
Yet in 2006, when ‘neural networks’ were rebranded as ‘deep learning’, it suddenly became the sole focus of AI research.This movement ignored all other techniques, investing money and publishing papers only if they involve DL. But focusing all our attention on this one path to the solution is like that first mountain in Chart 2; it might give good results for a while, but there’s a high chance of hitting a local optimum.
This has happened before — and the results were two AI winters.
In order to stop that from happening, we need to learn from the past, and save the future of AI by keeping diversity in AI research. Just like how we optimize objective functions in AI, we can’t trust on only the most promising current direction. We need to make sure we have enough “random restarts” while moving toward the optimum point. If you are using Classic AI, Shallow Learning, or Rule-Based Learning … keep up the awesome work!
— —
written by Ramtin Seraj and Wren Handman
