Predicting the Future: The Steamrollers and Machine Learning

Igor Carron
Sep 12, 2016 · 3 min read

Four years ago, I wrote that when one wants to predict the future, it is always a good thing to rely on the steamrollers, i.e. the exponential trends on which we can surf to predict the future because they are highly efficient. Let see how this trend continues.

From genome.gov

Since 2012, genome sequencing reached a second inflection point but I still expect a third inflection point due to the use of the nanopore sequencing technology as we had mentioned then. So we are good.

For Moore’s law it is a little more complicated. While the number of transistors do increase (but probably not as fast a Moore’s law anymore. In this most recent graph, the last five year show a slight inflection point) the performance is beginning to stall. Dennard scaling stopped around 2005.

From HPC no more free lunch

and while the number of transistors increased, we are beginning to see a decrease in the use of that silicon. This last trend has a name: Dark Silicon. Bill Dally, whom we had invited at the now-not-gonna-happen NIPS workshop said the following three years ago:

“The number of transistors is still going up in a very healthy way,” Dally contends, “We’re still getting the Moore’s law increase in the number of devices, but without Dennard scaling to make them useful, all the things we care about — like clock frequency of our parts, single thread performance and even the number of cores we can put in our parts — is flattening out, leading to the end of scaling as we know it. We’re no longer getting this free lunch of technology giving us faster computers.”

This is a problem that needs to be solved because our 21st century economy hinges on continued computational progress. With the end of Dennard scaling, all computing power becomes limited, and performance going forward is determined by energy efficiency. As process technology can no longer be counted upon for exponential advances, the focus must shift to architecture and circuits, according to Dally.

Since Machine Learning has become the main driver in applications, it has become a target of interest for the silicon industry. In a way, one of the lessons of that story is for architecture of Machine Learning algorithms to be a better fit for silicon architectures. No wonder we recently saw the rise of Google’s TPU, the numerous NVIDIA platforms like the DGX 1, the recent acquisition of silicon-machine learning companies like Nervana and Movidius, the effort of Baidu in the GPU clusters realm…

Let me finish this short history of the times we live in by pointing out that while Silicon will become more efficient because Machine Learning algorithms will be used as a constraint, making sense of the world in the future will still require new technologies. From the recently released report produced by the Semiconductor Industry Association and the Semiconductor Research Corporation entitled “Rebooting the IT Revolution: A Call to Action”, one can read that given the data explosion we are predicting, computing in 2040 may become problematic. From Appendix A Data Explosion Facts , A4. Total Effective Capacity to Compute Information in 2040:

For this benchmark energy per bit, computing will not be sustainable by 2040, when the energy required for computing will exceed the estimated world’s energy production. Thus, radical improvement in the energy efficiency of computing is needed.

And according to the authors of “Moore’s law: the future of Si microelectronics

The time frame to implement a radically new device is estimated to be ∼30 years.

We need to find less than radically new devices and better algorithms to make sense of this upcoming Miller’s wave in 2025. It’s time to get cranking.

Emergent // Future

Exploring frontier technology through the lens of artificial intelligence, data science, and the shape of things to come

Igor Carron

Written by

CEO LightOn , Co-organizer Paris Machine Learning Meetups #MLParis . Runs the Nuit-Blanche blog.

Emergent // Future

Exploring frontier technology through the lens of artificial intelligence, data science, and the shape of things to come

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade