AI Index Index

Summarizing a summary of the state of AI

Published in

Paper Club

5 min readDec 13, 2017

Earlier this month a team made up of and sponsored by Stanford, MIT, OpenAI, Google, Microsoft, etc. released the AI Index, a first-of-its-kind deep-dive into the trends in the Artificial Intelligence community.

It was a much-needed, sometimes surprising, mostly reassuring look at the state of ML/AI in 2017. I’m going to summarize some of my favorite parts and themes, but I would encourage you to read the report in its entirety here. The data itself runs for 40 or so pages of size 16 font with plenty of whitespace and lots of pictures (very refreshing after reading academic papers for months!); it is a casual half-hour skim.

Unfortunately the committee’s request for my inclusion in the Expert Forum got lost in the mail, so I’ve been forced to publish my more subjective thoughts on Medium here.

Volume of Activity (how much)

Growth rate in publishing of papers in AI is 5x the growth rate in publishing of papers in all other fields since 1996, and 1.5x the growth rate in publishing of computer science papers.
Interest in deep learning has not necessarily revived interest in other types of AI, as evidenced by conference attendance; the AAAI (Association for Advancement of AI) conference, which is typically associated more with symbolic reasoning, has seen attendance drop from ~5000 in the 1980’s to ~1500 today, while NIPS has gone from ~500 attendees in 2000 to ~5000 today.
Annual VC investment in startups was at ~$3.5B last year, following a huge spike earlier this decade.

Sentiment analysis of media coverage of AI shows that positive classifications went way up in 2016.

Technical Performance (how good)

State-of-the-art AI performance has passed human-level performance in object detection (in 2015) and speech recognition (in 2016).
State-of-the-art AI performance still lags behind human-level performance in textual (AI 79% accuracy, human 83% accuracy) and visual question answering (AI 68% accuracy, human 83%).

Derivative Measures (how trends relate)

This is one of my favorite graphs in the report. It illustrates that researchers drove interest in AI in the early 2000s, which prompted higher higher enrollment in AI-related classes around 2010, and finally investor money hopped on the AI train earlier this decade. This fits nicely into a narrative about the progress of AI this century.

Towards Human-Level Performance? (how close to human-level performance)

The biggest issue with current implementations of neural networks is that they are very narrowly trained and thus performance is quite fragile. If the target task is moved even slightly from the expectation formulated during training, performance can fall off a cliff. Nevertheless, AI has been able to achieve several real-world milestones against human competition, with more barriers seemingly falling every month.

What’s Missing?

We need more standardization for both benchmarks and research practices across the industry in order to be able to trust and compare published results.
Almost all of the data in this report comes out of the US, even though China is neck-and-neck in terms of progress in AI and countries like the UK and Canada are experiencing huge surges of interest in the field as well.
Echoing a sentiment in the tech industry at large, diversity and inclusion are ongoing issues in AI, and it’s a red flag that there isn’t any available data on this.
This report covers VC investment, but government and corporate investment into AI likely have even larger impacts. There’s no data on these types of investment.
Ethics in AI is an ongoing conversation that can’t be ignored: safety, predictability, fairness, privacy, etc.

Expert Forum

(I’m summarizing folks’ thoughts here. Stuff will inevitably get lost in translation, so plz don’t draw any conclusions from this without reading their entire bit in the forum)

Although the authors were quite thorough with What’s Missing, the most common theme in the Expert Forum was emphasizing or piling on to this section. Particular areas of concern were ethics, safety, diversity, and inclusion, and US-centricity.
Kai-Fu Lee (Sinovation Ventures) notes that people in China use their phones to pay for goods 50x more than Americans and have 10x higher online food delivery volume despite having 3x the population, indicating a potential goldmine of data collection since the people are so much more engaged in technology. He also brings up the China State Council’s “Next Generation Artificial Intelligence Development Plan”, which is putting the full weight of the government behind an effort to make China the global leader in AI by 2030.
Andrew Ng (Coursera, Stanford) compares AI to electricity in terms of its impact across almost every industry in the world.
Daniela Rus (MIT) highlights the potential democratization of education through AI. She sees a world where any child anywhere in the world can receive an AI-designed curriculum online that adapts to their progress and learning style.
Sebastian Thrun (Stanford, Udacity) compares AI to the steam engine in its ability to automate repetitive work and free people to dedicate more time and energy to make advancements in society. This is both a blessing and a potential curse, as people whose jobs will be displaced by AI must adapt to survive.
Michael Woolridge (Oxford) calls AI a bubble, but he expects a gentle deflation over the next few years as steady progress replaces explosive progress, rather than an all-at-once implosion.