Artificial Intelligence Index Annual Report 2017 - Takeaways for a Millennial

6 min readDec 2, 2017

A recent comprehensive report sponsored by Google, Microsoft and ByteDance was aggregated by the Artificial Intelligence Index group, a project within the Stanford 100 Year study on AI. The AI Index is an initiative to track, collate, distill and visualize data relating to artificial intelligence with the mission statement of :

“grounding the conversation about AI in data.”

It aspires to be a comprehensive resource of data and analysis to rapidly develop intuitions about the complex field of AI. The full original report aggregates a diverse set of data, makes that data accessible, and includes discussion about what is provided and what is missing. This is a version is curated in context of college students.

Overview (In for a surprise.)
Volume of Activity (The hype is real.)
Performance Metrics (Told you the hype is real.)
Derivative Measures (Now you’re talking.)
Towards Human-level Performance
Remarks from the Experts
Ending Notes

Overview

There is no overview in a short and sweet version.

Volume of Activity

Academia

The number of AI papers produced each year has increased by more than 9x since 1996.

Course enrollment in introductory AI & ML courses has substantially increased. (11x in Stanford, guess why?)

The report also mentioned the caveat of this segment of data limited to the specific silver of the academia landscape.

Industry

The number of active US startups developing AI systems has increased 14x since 2000.

Again, the nature of research and development and the technology backing the venture doesn’t provide a fine line of division. And it gets more tricky as the data covers active ventures, foregoing the attrition.
Annual VC Investment into US companies developing AI has increased 6x since 2000.
The share of jobs requiring AI skills in the US has grown by 4.5x since 2013, with Canada and UK at more than 11x and 8x respectively. Again, this data was taken from two online job listing portals, Indeed and Monster, and is perhaps limited to the tags that carry along openings. (Basically, the plausible conclusion is that the hype is real.)
The skills breakdown revolve around — Machine Learning, Deep Learning, NLP, Computer Vision, Speech Recognition — in that order.
Sentiment Analysis of popular media articles that contain the term “Artificial Intelligence” classified as positive or negative.

Performance Metrics

Vision

The performance of AI systems on the Large Scale Visual Recognition Challenge Competition has surpassed human accuracy. Error rates have dropped from 28.5% to 2.5% since 2010. Now before you consider joining Anthony Levandowski and his cult, we are far from it. The technological advances and orientation of community are to be majorly credited for the improvement.
The performance of AI systems on a task to give open-ended answers to questions about images.

(Phew!)

Natural Language Processing

Parsing, which is the performance of AI system on a task to determine the syntactic structure of sentences. Basically making sense of the various text inputs at various levels of abstraction.

For non-field folks : Notice the slack in the ‘sentences of all length’ because of inability to hold context in long conversations. There’s a short film starring Thomas Middleditch from Silicon Valley among other, created entirely scripted by an AI system highlighting this fact.
The performance of AI systems on a task to find the answer to a question from a document.

In these contexts the performance of AI systems on the task to recognize speech from phone call audio has recently reached human intelligence.

Derivative Measures

An attempt to derive hypothesis from largely unrelated datasets. (Which makes it interesting.)

Academia-Industry Dynamics

What they did : Normalized each of the data points from AI paper publishing, enrollment in introductory AI and related courses and VC investments in AI startups to the year 2000 values.

Hypothesis : The data shows that, initially, academic activity (publishing and enrollment) drove steady progress. Around 2010 investors started to take note and by 2013 became the drivers of the steep increase in total activity. Since then, academia has caught up with the exuberance of industry.

Towards Human-Level performance

Tasks for AI systems are often framed in narrow contexts for the sake of making progress on a specific problem or application. While machines may exhibit stellar performance on a certain task, performance may degrade dramatically if the task is modified even slightly.
Yet, it is important to note super-human performance in specific task contexts in recent times. I’m assuming most of us to be aware of the astounding victory of AlphaGo over Lee Sedol. Other areas are Skin Cancer classification, Speech Recognition on switchboard, Poker and Pac-Man.

The authors were open about the limitations of the dataset. Some known ones being lack of international coverage, diversity and inclusion, absence of certain fields such as automotive, finance, education to name a few

Remarks from the Experts

“Countries with more sensible AI policies will advance more rapidly, and those with poorly thought out policies will risk being left behind.” — Andrew Ng, Coursera.
“On a global scale, AI will help us generate better insights into addressing some of our biggest challenges: understanding climate change by collecting and analyzing data from vast wireless sensor networks that monitor the oceans, the greenhouse climate, and the plant condition; improving governance by data-driven decision making; eliminating hunger by monitoring, matching and re-routing supply and demand, and predicting and responding to natural disasters using cyber-physical sensors.” — Daniela Rus, MIT.
“There is, clearly, an AI bubble at present; the question that this report raises for me is whether this bubble will burst (cf. the dot com boom of 1996–2001), or gently deflate; and when this happens, what will be left behind? My great fear is that we will see another AI winter, prompted by disillusion following the massive speculative investment that we are witnessing right now.” — Michael Wooldridge, Oxford.

Ending Notes

In the time to come, AI’s impact on the progress of the world is going to involve a lot of decisions it is imperative for everyone to make an informed one.
Would be great to have a similar report for the Indian AI Ecosystem.
Sit tight, its a bumpy ride.

Great acknowledgement for the people behind the project : Yoav Shoham, Ray Perrault, Erik Brynjolfsson, Jack Clark and Calvin LeGassick.

You can go through the entire document (101 pgs) here. For data-savvy crowd, here is the link to some of their raw data for running your independent analysis.

Note : I was in no capacity connected to the project and the sole purpose of the article is to curate relevant information.

Hope this helped.