A.I. Hardware: the opportunity for a fast, persistent memory

Hardware is the key battlefield for AI

Software has been the star of high tech over the past few decades, defining the game-changing innovations that defined the era. Although innovations in chip design have practically enabled all of these next-gen devices, semiconductor companies have only captured a small share of the total value in all this technology — about 25% of the value in PCs and a meager 15% in Mobile according to McKinsey. But the story for semiconductor companies will be different in the race for AI. Hardware is becoming the main performance bottleneck and solutions to the bottlenecks become differentiators. That’s the reason why leading Internet players– such as Google, Facebook, Amazon, Apple — are rushing to become silicon designers in search of a hardware competitive-edge. For example, Amazon has developed a custom Arm-based server processor, increasing the competiveness, by lowering their costs 45%. The value at stake is massive. According to McKinsey, the AI semiconductor market should reach $65B by 2025, growing at 15% CAGR. Silicon startups, for the first time in decades, are now well-positioned to capitalize on this opportunity. The agility of small cross-functional teams is well suited for learning fast and developing innovative hardware solutions. VCs have also realized this unique opportunity and invested a record $9.3B invested in AI startups in 2018.

AI Poses a Major Computing Challenge

AI is rapidly building a connected world where sensing, computing and communications are incorporated into tens of billions of devices that monitor their environments, make decisions and send information to the cloud. One trillion new IoT devices will be produced by 2035 according to Arm. These devices will generate an explosion of data to be processed. By 2022, according to Cisco, global mobile data traffic will reach more than 930 exabytes annually, 46% ’17-’22 CAGR. An autonomous vehicle in 2023 will generate 4Tb per day, compared to the ~1Gb per day currently generated by an individual. For the first time in history, machines are creating more data than humans, a trend that will accelerate as AI expands.

But data is useless unless it is effectively captured, manipulated, extracted and exploited. This exponential growth of data creation is creating a massive computing and storage challenge to extract information efficiently in terms of performance, energy, and cost.

At every level of the computing infrastructure — from edge to data center and back — energy consumption is becoming an ominous hurdle. At the edge, we often need ultra-low-power solutions that can run on battery or even harvested energy. In these cases, compute, memory and AI will be local. And while the power consumption of most IoT edge devices is low, the total energy consumption is staggering simply due to the massive number of them.

Consider a simple IP camera for home security. It consumes only 5 to 8 watts, but in 2020 all IP cameras combined will consume more power than the energy generated by a standard power plant in the United States. In the data center, projected by Applied Materials to consume 10% of all electricity world-wide by 2025, the key constraint is the amount of energy the power company can deliver to the building, rather than what the data center costs. And the problem will get worse, and get worse exponentially. It is estimated that training a single AI model can emit as much as carbon as five combustion engine automobiles over their operating lifetime.

Gary Dickerson, CEO of Applied Materials, defines the AI grand challenge as the need to improve compute performance per watt by 1,000x. He highlights that the growth of AI demands immediate development of more energy-efficient computing paradigms.

AI-specific Hardware Are Required

The pending crisis is that the evolutionary balance between cost, performance, and energy efficiency that fueled the computer industry for more than 40 years has stalled and can no longer match the exponential growth of data and compute: transistors fundamentally are not shrinking due to the ending of Moore’s Law, power is limiting what can be put on a chip due to the end of Dennard scaling, and parallelism, the number of processors per chip, is approaching its limits due to Amdahl’sLaw.

Thus, the rapid improvements in processing that we have enjoyed with the Moore’s law era must now come through innovations in computer architecture and hardware instead of semiconductor process improvements. The current performance bottleneck is in a component of computing that most people rarely consider: memory.

Read more: Memory holds the keys to AI adoption