Why the ‘AI revolution’ is really a deep learning revolution

A simple quantitative comparison of trends in AI journalism vs AI research

We often hear that there’s been an explosion in artificial intelligence (AI) research, that AI is advancing a breakneck speed. Here is a histogram of Guardian headlines containing “AI” collected via the Guardian’s Open Platform when I searched for “technology”. I counted 60 AI headlines from 2017. I knew AI would be trending but I was surprised at how strong this trend is.

Figure 1: Percentage of Guardian Technology Headlines with “AI” in the title

When we look at computer science publications in research one might be tempted to think there is a similar trend in new and emerging AI techniques but this isn’t exactly the case. There is one big obvious new kid in town, deep learning. Below we shall see that since 2012 deep learning has gone from an unused term to a phrase that dominates research in the fields of artificial intelligence, natural language processing, computer vision and machine learning. Deep learning is trending in academic publications just like AI is trending in newspapers.

Research is not going through an AI revolution, it is a deep learning revolution.

The distinction between AI and deep learning is important. The term AI is evocative of an all encompassing technology that can be applied to everything and anything. Deep learning is an emerging machine learning technique that has strengths and weaknesses, just like any other technology. It has completely transformed some problems such as image recognition and language translation but in some other domains it is not as applicable.

Below I use data from arXiv to suggest media interest in AI is driven by advances in deep learning. To some people this may seem like an obvious conclusion, but remember, it is not necessarily obvious to everybody. Moreover, it is important to make sure our intuitions are backed by data and observations. Even if this isn’t news to you, I suspect you will be interested to see just how strong this trend in deep learning is.

All the results in this article can be reproduced using this GitHub repository.

The Dataset

arXiv is a free and open repository of journal preprints that have been approved for publication after moderation. It is where many computer scientists share their research to provide free and fair access to their work.

Let’s take the titles of arXiv computer science publications tagged with any of the following categories which are relevant to AI (see the full list of tags here):

• Artificial Intelligence
• Computation and Language (Computational Linguistics and Natural Language and Speech Processing)
• Computer Vision and Pattern Recognition
• Machine Learning
• Neural and Evolutionary Computation

I took the titles of all the publications with these tags and the year of publication. Then I simply looked for which words which have fluctuated over time. Again, all the details are on GitHub

It’s worth noting at this point, unlike newspaper headlines, the term AI is rarely used, if at all, in research. Only 138 of a total of 43244 articles used “AI” or “artificial intelligence” in the title. Too few titles to see any meaningful trends.

Results

Fuzzy logic and logical programming were staples through the late nineties and early 21st century. In figure 2 we can see a steady share of arXiv publications containing the word “logic” in the title which has diminished in recent years.

Figure 2: Percentage of selected arXiv publications with “logic” in the title

From figure 3 evolutionary algorithms and evolutionary programming don’t appear to be trending in particular either.

Figure 3: Percentage of selected arXiv publications with “evolutionary” in the title

Consider figures 2 and 3 as baselines. Although they have their own trends there is nothing especially interesting here about the years 2012–2014, the years when AI started trending in journalism. Keep this in mind when we explore data connected with deep learning.

When we histogram arXiv papers containing the word “deep” there is a dramatically different story (figure 4). Just like with the guardian headlines I suspected there would be a trend around deep learning, but I didn’t expect to see something as dramatic as this.

Figure 4: Percentage of selected arXiv publications with “deep” in the title

Academic publications containing “deep” are trending in a suspiciously similar way to newspaper headlines containing “AI”.

The following ten words and bi-grams (all strongly connected to deep learning) have seen the largest increase in usage since 2012 according to the arXiv dataset:

“deep learning” “detection” “convolutional neural” “image” “neural networks” “convolutional” “neural” “networks” “learning” “deep”

and the following 10 words have dropped most in usage:

“logic” “systems” “programming” “algorithm” “information” “constraint” “algorithms” “theory” “language” “complexity”

When we take a set of phrases connected to deep learning such as “deep” or “adversarial” or “convolutional” we get an even clearer picture as in figure 5. It appears that in excess of 1 in 5 of the selected publications could be concerned with deep learning.

Figure 5: Percentage of selected arXiv publications with either “deep”, “adversarial” or “convolutional” in the title

Conclusions

The AI research landscape has transformed since 2012. Post 2012 the landscape has become much more concentrated on one topic, deep learning. Following the incredibly rapid upward trends demonstrated above deep learning is on course to saturate AI research in the next few years, if this trend is maintained.

The apparent correlation between AI in the news and deep learning in research suggests that deep learning is driving media interest. No doubt deep learning applications such as alpha go, style transfer networks and image translation have understandably captured popular imagination. Advances in deep learning are best reflected in products like Google Translate, Siri, Alexa and visual search tools, yet deep learning is not a silver bullet.

While deep learning will continue to make huge strides in some areas it is less likely to have impact in others, most obviously those where it not possible to collect gigantic quantities of training data. Nor is deep learning able to imitate human behaviour in general, even if it has made progress on narrow problems commonly associated with humans such as language translation.

If we do not understand the strengths and weaknesses of deep learning and and the distinction between deep learning and AI we risk a gap between expectation and reality and between researchers and journalists.

I’ll leave you with 5 Headlines and 5 arXiv publications, randomly selected to give a flavour of the dataset

Generative learning for deep networks

Would you bet against sex robots? AI ‘could leave half of world unemployed’

Algorithms: AI’s creepy control must be open to inspection

AI, self-driving cars and cyberwar — the tech trends to watch for in 2017

On the interplay of network structure and gradient convergence in deep learning

The Robot Will See You Now review — it appears even therapists could lose their jobs to AI

MUFold-SS: Protein Secondary Structure Prediction Using Deep Inception-Inside-Inception Networks

Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networks

AI scientists want to make gods. Should that worry us?

Harvesting Discriminative Meta Objects with Deep CNN Features for Scene Classification