10 Alarming Predictions for Deep Learning in 2018
I’ve got this ominous feeling that 2018 could be the year when everything just changes dramatically. The incredible breakthroughs we saw in 2017 for Deep Learning is going to carry over in a very powerful way in 2018. A lot of work coming from research will be migrating itself into everyday software applications.
As I’ve done last year, here are my predictions for 2018. (For reference, here are my 2017 predictions and a recap)
- A Majority of Deep Learning Hardware Startups will Fail
Many Deep Learning hardware startup ventures will begin to finally deliver their silicon in 2018. However, these startups will mostly be busts because they will forget to deliver good software to support their new solutions. These firms have hardware as their DNA. Unfortunately, in the DL space, software is just as important. Most of these startups don’t understand software and don’t understand the cost of developing software. Thus, these firms may deliver silicon, but nothing will ever run on them!
The low hanging fruit that employs systolic array solutions has already been taken, so we won’t have the massive 10x performance upgrade that we found in 2017. Researchers will start using these tensor cores not only for inference, but also to speed up training.
Intel’s solution will continue to be delayed and will likely disappoint. The record shows that Intel was unable to deliver on a mid-2017 release and it’s anybody’s guess when they will ever deliver. It’s late and it’s going to be a dud.
Google will continue to surprise the world with its TPU developments. Perhaps Google gets into the hardware business by licensing their IP to other semiconductor vendors. This will make sense if they continue to be the only other real player in town other than Nvidia.
2. Meta-Learning will be the new SGD
A lot of strong research in Meta-learning appeared in 2017. As the research community collectively understands meta-learning much better, the old paradigm of Stochastic Gradient Descent (SGD) will fall in the wayside in favor of a more effective approach that combines both exploitive and exploratory search methods.
Progress in unsupervised learning will be incremental, but it will be primarily driven by Meta-learning algorithms.
3. Generative Models drives a New Kind of Modeling
Generative models are going to find themselves in more scientific endeavors. At present, most research is performed in generating images and speech. However we shall see these methods being incorporated in tools for modeling complex systems. One of the areas where you will see more activity is in the application of Deep Learning on economic modeling.
4. Self-Play is Automated Knowledge Creation
AlphaGo Zero and AlphaZero’s “learn from scratch self play” is a quantum leap. In my opinion, it is of the same level of impact as the discovery of Deep Learning. Deep Learning discovered universal function approximators. RL self-play discovered universal knowledge creation. DeepMind doesn’t have a word for this yet, but another research group employs the dual process model of cognition to explain it and calls it “ExIt”.
Do expect to see a lot more advances related to self-play.
5. Intuition Machines will Bridge the Semantic Gap
This is my most ambitious prediction. The semantic gap between intuition machines and rational machines will be bridged (if it is not already been bridged). Dual Process Theory (the idea of two cognitive machinery, one that is model-free and the other that is model-based) will be the more prevalent conceptualization of how new AI should be built. The notion of Artificial Intuition will be less of a fringe concept and be a more commonly accepted idea in 2018.
6. Explainability is Unachievable. We will just have to Fake It
There are two problems with explainability. The more commonly known problem is that the explanations have too many rules that a human cannot possibly grasp. The second problem, which is less known, is that a machine will create concepts that will be completely alien and defy explanation. We already see this in the strategies of AlphaGo Zero and Alpha Zero. Humans will observe that a move is unconventional, however, they simply may not have the capacity to understand the logic behind the move.
This, in my opinion, is an unsolvable problem. What will happen instead is that machines are going to become very good at ‘faking explanations’. In short, the objective of explainable machines is to understand the kinds of explanations that a human can be comfortable with or will understand in an “intuitive” level. However, a complete explanation will, in a majority of cases, be completely inaccessible to humans.
Progress in explainability in Deep Learning will be made by creating ‘fake explanations’.
7. Deep Learning Research Information Deluge
2017 was already difficult for people following Deep Learning research. The number of submissions in the ICLR 2018 conference was around 4,000 papers. A researcher would have to cover 10 papers a day just to catch up with this conference.
The problem is worsened in this space because here, the theoretical frameworks are all works in progress. To make progress in the theoretical space, we need to seek out more advanced mathematics that can give us better insight. This is going to be a slog simply because most Deep Learning researchers don’t have the right mathematical background to understand the complexity of these kinds of systems. Deep Learning needs researchers coming from complexity theory, however there are very few of these kinds of researchers.
As a consequence of too many papers and poor theory, we are simply left with the undesirable state of alchemy that we find ourselves today.
What is also missing is a general roadmap for AGI. The theory is weak, therefore the best we can do is create a roadmap with milestones that relate to human cognition. We only have framework that originate from speculative theories coming from cognitive psychology. This is a bad situation since the empirical evidences coming from these fields are spotty at best.
Deep Learning research papers will perhaps triple or quadruple in 2018.
8. Industrialization via Teaching Environments
The road to more predictable and controlled development of Deep Learning systems is through the development of Embodied Teaching environments. I have discussed this in a little bit more detail here and here. If you want to find the crudest form of teaching technique, then one only has to look at how Deep Learning networks are trained. We are all due a lot more progress in this area.
Expect to see more companies revealing their internal infrastructure as to how the deploy Deep Learning at scale.
9. Conversational Cognition
The way we measure progress towards AGI is antiquated, thus a new kind of paradigm that addresses the dynamic (i.e. non-stationary) complexity of the real world is demanded. We shall see more coverage in this new area in the coming year. I will be speaking about this new Conversational Cognition paradigm in Amsterdam ( March 1–2, Information Energy 2018).
10. Ethical Use of Artificial Intelligence
The demands for more ethical use of Artificial Intelligence will increase. The population is now becoming more aware of the disastrous effects of unintended consequences of automation run amok. Simplistic automation that we today find in Facebook, Twitter, Google, Amazon etc. can lead to unwanted effects on society.
We need to understand the ethics of deploying machines that are able to predict human behavior. Facial recognition is one of the more dangerous capabilities that we have at our disposal. Algorithms that can generate media that is indistinguishable from reality is going to become a major problem as well. We as a society need to make the rapid transition to begin demanding AI to be used solely for the benefit of society as a whole and not as a weapon to increase inequality.
Expect to see more conversation about ethics in the coming year. However, don’t expect any new regulations. Policy makers are still years behind the curve in understanding the impact of A.I. to society. I don’t expect them to stop playing politics and start addressing the real problems of society. The U.S. population have been victims of numerous security breaches, yet there is zero new legislation or initiatives to address this serious problem. So don’t hold your breath on our dear leaders suddenly discovering new found wisdom.
Prepare for Impact!
That’s all I have for now. 2018 is going to a major year and we all better strap on our seat belts and prepare for impact!
