Revolution in Deep Learning
Deep learning has emerged as the most important computational workload of our generation. In the past five years, Artificial Intelligence (AI) has risen from obscurity to top-of-mind awareness because of advances in deep learning. Tasks that historically were the sole domain of humans are now routinely performed by computers at human or superhuman levels.
Deep learning is profoundly computationally intensive. A recent report by OpenAI showed that, between 2012 and 2018, the compute used to train the largest models increased by 300,000X. In other words, AI computing is growing at a rate that is 25,000X faster than Moore’s law at its peak. AI compute demand is doubling every 3.5 months.
This voracious demand for compute means AI is constrained not by applications or ideas, but by the availability of compute. Testing a single new hypothesis — training a new model — takes weeks or months and can cost hundreds of thousands of dollars in compute time. This has slowed innovation to a crawl. Google, Facebook, and Baidu, among others, have noted that long training time is the fundamental impediment to AI progress; that many important ideas are ignored simply because these models take too long to train.
To meet the growing computational requirements of AI, Cerebras has designed and manufactured the largest chip ever built. The Cerebras Wafer Scale Engine (WSE) is 46,225 millimeters square, contains more than 1.2 trillion transistors, and is entirely optimized for deep learning work. By way of comparison, the WSE is more than 56X larger than the largest graphics processing unit, containing 3,000X more on chip memory and more than 10,000X the memory bandwidth.
Bigger is better in Artificial Intelligence compute. Larger chips process information more quickly and produce answers in less time. The WSE reduces the time it takes to do the most complicated AI work from months to minutes.
Source: The above has been taken from the Cerebras white paper “Cerebras Wafer Scale Engine: An Introduction”. You can find the full white paper for a deeper dive into the technology here.
If you found value in this article, please “clap” (up to 50 times).
This article is part of our Tokyo FinTech Publication, please follow us to read more from our writers, like hundreds of readers do every day.
Should you live in Tokyo, or just pass through, please also join our Tokyo FinTech Meetup. In any case, our LinkedIn page, Facebook page and our Instagram account are there for you as well.
