5 Ways to Track AI Impact, Challenges
New report measures the advances, describes the ethics of a maturing technology
By Irving Wladawsky-Berger
If anyone still has doubts about the impact AI is having on real-world economies, the 2022 AI Index firmly puts the debate to rest.
On March 16, Stanford’s Institute for Human-Centered Artificial Intelligence (HAI) released the report, its fifth annual study on the impact and progress of AI. The comprehensive study was led by an interdisciplinary group of experts from across academia and industry, and highlights the rapid rate of technical and marketplace AI advances, as well as the growing ethical and regulatory concerns associated with AI.
“2021 was the year that AI went from an emerging technology to a mature technology — we’re no longer dealing with a speculative part of scientific research, but instead something that has real-world impact, both positive and negative,” said Jack Clark, co-chair of the AI Index.
“This year’s Index tells us that AI is being integrated into the economy and the effects of it are beginning to go global across research, deployment, and even funding.”
Two weeks ago I attended an online HAI seminar where Clark gave this excellent presentation on the AI Index. The report is organized into five chapters: Technical Performance, AI Ethics, R&D, Economy and Education, and Policy and Governance. Let me summarize the key findings of each chapter.
- Technical Performance
- AI has become more affordable and higher performing, leading to more widespread commercial adoption. “Since 2018, the cost to train an image classification system has decreased by 63.6%, while training times have improved by 94.4%.” Lower training costs and faster times are seen across other AI tasks including recommendation engines, object detection, and language processing.
- Top technical benchmarks have increasingly relied on very large training data. Most leading-edge AI benchmarks have been trained with extra large data sets. This implicitly favors large companies with access to vast amounts of data.
- AI already exceeds human performance levels on basic reading comprehension benchmarks, such as SuperGlue and SQuAD, but has not yet has not mastered complex language tasks.
AI is still unable to achieve human performance on complex linguistic tasks such as aNLI, but the difference is narrowing.
- Interest is rising in specific computer vision tasks like medical imaging segmentation and masked-face identification, as AI research is increasingly focused on more real-world applications. “For example, only three research papers tested systems against the Kvasir-SEG medical imaging benchmark [a gastrointestinal polyp segmentation application] prior to 2020. In 2021, 25 research papers did.”
- AI research is making progress in general reinforcement learning. Over the past decades, AI systems have been able to master highly focused reinforcement learning tasks, such as chess, Go, and video games. “However, in the last two years AI systems have also improved by 129% on more general reinforcement learning tasks (Procgen) [a reinforcement learning benchmark] in which they must operate in novel environments. This trend speaks to the future development of AI systems that can learn to think more broadly.”
- Robotic arms are becoming cheaper making robotics research more accessible and affordable. “An AI Index survey shows that the median price of robotic arms has decreased by 46.2% in the past five years — from $42,000 per arm in 2017 to $22,600 in 2021.”
2. AI Ethics
- AI ethics is now everywhere. The rapid deployment of real-world AI applications has led researchers and practitioners to reckon with real-world harms that reflect and amplify human social biases and generate false information. These include “commercial facial recognition systems that discriminate based on race, résumé screening systems that discriminate on gender, and AI-powered clinical health tools that are biased along socioeconomic and racial lines.”
- Large language models are more capable than ever, but also more biased. However, larger models are also more prone to reflecting toxicity and bias from their training data. “280 billion parameter model developed in 2021 shows a 29% increase in elicited toxicity over a 117 million parameter model considered the state of the art as of 2018.”
- Research on fairness and transparency in AI has exploded. Since 2014, there’s been a fivefold increase in related publications at ethics-related conferences.
Algorithmic fairness and bias is becoming a mainstream research topic with wide-ranging implications.
“Researchers with industry affiliations contributed 71% more publications year over year at ethics-focused conferences in recent years.”
- Multimodal language-vision models may lead to increased biases. Language-vision models have set new records on tasks like image classification and the creation of images from text descriptions, “but they also reflect societal stereotypes and biases in their outputs — experiments on CLIP showed that images of Black people were misclassified as nonhuman at over twice the rate of any other race.”
3. Research and Development
- Publications. The total number of AI publications in the world has grown from around 162,500 in 2010 to 334,500 in 2021, with journal articles (51.5%) and conference papers (21.5%) as the two biggest categories. China was the leader in both journal articles (31% of the total), and conference, papers (28%), followed by the European Union and UK (19% in each category), and the U.S. (13.7% and 17% respectively).
- Cross-Country Collaborations. “Despite rising geopolitical tensions, the United States and China had the greatest number of cross-country collaborations in AI publications from 2010 to 2021, increasing five times since 2010. The collaboration between the two countries produced 2.7 times more publications than between the United Kingdom and China — the second highest on the list.”
- Cross-Sector Collaboration. The education sector had the most AI publications (59.6%) in 2021, followed by nonprofits (11.3%), private companies (5.2%), and government (3.2%). “From 2010 to 2021, the collaboration between educational and nonprofit organizations produced the highest number of AI publications, followed by the collaboration between private companies and educational institutions and between educational and government institutions.”
- Patents. Over 30 times more patents were filed in 2021 than in 2015, a compound annual growth rate of 77%. In 2021, China filed over half (51.7%) of the world’s patents, followed by the U.S. (16.9%) and the EU and U.K. (3.9%).I
4. Economy and Education
- Investment. In 2021, private investment in AI totaled around $93.5 billion, more than double the $46 billion invested in 2020 and over 10 times the $9 billion invested in 2015. However, the number of newly funded AI startups dropped from 1051 companies in 2019 and 762 in 2020, to 746 in 2021 — a reflection of the increasing adoption of AI by larger, more mature companies.
In addition, there were 15 funding rounds of $500 million or more in 2021 compared to four in 2020.
- Data management, processing, and cloud received the greatest amount of private AI investment in 2021, 2.6 greater than in 2020. It was followed by medical and healthcare, fintech, audio-visual, semiconductor, industrial automation, retail, and fitness and wellness.
- In 2021, the U.S. led the world in overall private investment in AI companies ($52.9 billion), followed by China ( $17.2 billion), the European Union ($6.42), the U.K. ($4.65 billion), and Israel ($2.4 billion).. The US also led in the number of newly funded AI companies (299) in 2021, followed by China (119), the EU (106), the U.K. (49), and Israel (28).
- Education. In 2020, AI/ML was the most popular specialty (21%) of U.S. PhD graduates in Computer Sciences, while Robotics/Vision was the 6th most popular specialty (6.3%). Between 2010 and 2020, the number of CS PhD graduates in the U.S. with AI/ML and robotics/vision specialities grew by 72% (161 to 277) and 51% (55 to 83) respectively.
- In 2020, 60% of new AI/ML PhDs in the U.S. went to industry, 24% went to academia, 2% went to government, and 12% took positions outside the U.S. The percentage of PhDs going into industry vs academia has increased substantially over the past decade, from roughly the same percentage in 2010 to 2.5 times bigger in 2020. 60.5% of new AI/ML PhDs were international students, 14% of which took jobs outside the US.
5. Policy and Governance
- Legislation: There was a sharp increase of the number of proposed AI-related bills in the U.S. Congress between 2015 and 2021, but only 2% ultimately became law. “[T]he current congressional session (the 117th) is on track to record the greatest number of AI-related mentions since 2001, with 295 mentions by the end of 2021, half way through the session, compared to 506 in the previous (116th) session. …State legislators in the United States passed one out of every 50 proposed bills that contain AI provisions in 2021, while the number of such bills proposed grew from two in 2012 to 131 in 2021.”
Overall, the 2022 AI Index shows that after decades of promise and hype, AI has finally become one of the defining technologies of our era. AI is already having an impact on real-world economies, and addressing its ethical and societal concerns is now a priority for governments around the world.
This blog first appeared April 15 here.