Why Neuromorphic Matters: Deep Learning Applications
Volume 78, October 1990, IEEE.
It was in this seemingly unremarkable publication that Carver Mead laid the rudimentary foundations of “Neuromorphic Electronic Systems.”
Okay, what in the world is Neuromorphic Electronic Systems?
When I first read this seminal paper more than a decade ago, I was a naïve engineering undergraduate at U.C. Berkeley.
What struck me most were these final words by Dr. Mead — a decorated Professor of Engineering and Computer Science at the renowned California Institute of Technology.
“I expect large-scale adaptive analog technology to permit the full utilization of the enormous, heretofore unrealized, potential of wafer-scale silicon fabrication.”
Before we dive in, I encourage you to first read the two other articles in this 3-Part Series: The VC’s Guide to Machine Learning and Creating Music by Machine Learning.
As you explore this subject, it can be easy to get lost in the forest, so keep this in mind as you’re learning about Neuromorphic Computing: it is simply the physical extension of Artificial Intelligence’s elusive goal of finding the optimal Representation of Reality.
An Impending Crisis in Machine Learning
Before we get to Neuromorphic Computing, let’s first understand the current state of technology. Almost every computer today is based on the “von Neumann architecture.”
Named after one of the greatest scientists of the 20th century — John von Neumann — the von Neumann architecture separates a computer’s computational cells (the CPUs) from its memory cells through a communication bus.
This hardware architecture was ultimately adopted because of the incompatibility between a silicon fabrication technology known as CMOS utilized in CPUs versus memory chips.
Indeed, the von Neumann architecture has driven the exponential growth of computing known as Moore’s Law for the past 50 years. However, this growth is unsustainable.
First, under the von Neumann architecture, power consumption and computing capability need to be balanced and cannot be achieved simultaneously.
Thus even as transistor sizes have shrunk below 10 nanometers, there is still a power consumption/computing tradeoff that imposes a limiting lower bound.
Second, there is a physical limit on how small we can manufacture transistors due to an effect known as quantum tunneling for transistors smaller than 1–3 nanometers, which we have reached.
Although Intel and AMD — the two largest CPU manufacturers — have tried to get around this using parallel processing (or simply connecting many existing CPUs together), the fundamental von Neumann bottleneck remains.
Finally, the nail in the coffin for Artificial Intelligence is that Machine Learning algorithms frequently move large amounts of data, a power- and memory-intensive task which is not scaleable for CPUs and GPUs operating under the von Neumann architecture.
This monumental problem has led to projects like SpiNNaker (Spiking Neural Network Architecture) and Graphcore developing Graph Processors, planting the seeds for Neuromorphic Computing.
Neuromorphic Computing for Deep Learning
There are three principal neuromorphic computing architectures.
Type A is the Distributed Computing Architecture (DNCA) which intersperses computing units (“neurons”) and memory units (“synapses”) in a distributed brain-like network.
Type B is the Cluster Neuromorphic Computing Architecture (CNCA), which is motivated by the brain’s sensory signals (somatic, tactile, auditory, visionary, olfactory, and gustatory), clustering different processing functions according to the Type A architecture.
Type C is perhaps the most promising as it addresses a fundamental problem of most machine learning agents (including the remarkable GPT-3 NLP agent developed by OpenAI), namely the capacity of AI for associative learning.
In this Associative Neuromorphic Computing Architecture (ANCA), signals are first processed in specialized regions akin to the Type B architecture. Additionally, the processed information is then coupled to each other to construct an associative neural network.
Okay, this all sounds promising.
But how do we actually build a neuromorphic chip?
Albert Einstein was once asked what was the secret to his genius, to which he answered:
“Look deep into nature and you will understand everything better.”
Indeed, it was perhaps by looking into nature that Leon Chua — Professor of Electrical Engineering at U.C Berkeley whom I had the privilege of taking one of his courses — discovered the memristor, an electronic device that mirrors the function of a human neuron’s synapse.
Long a topic of academic research, memristors have only recently gained attention with the remarkable accomplishments of Deep Learning (see AlphaGo, AlexNet, and Transformers).
As the Director of Tesla’s AI Research highlighted, the computational challenges of Deep Learning cannot be solved in the long-term by traditional von Neumann computers.
Especially for life-critical applications like Autonomous Driving, Deep Learning in the “cloud” or even through edge computing is not acceptable when reliability is paramount.
Without getting into the technical details of “3D Monolithic Integration,” below is an illustrative mapping of an Artificial Neural Network (ANN) to a 3D Neuromorphic Chip using memristors for the “synaptic layer.”
Okay, so why are we doing all this?
Let’s take a step back.
The human brain consumes just 20 Watts of electricity — less than a typical lightbulb — yet can outcompete OpenAI’s GPT-3 on any simple associative learning task.
It is estimated that it costs about $5 million to train the 175 Billion parameters of GPT-3.
Thus, it becomes clear that current (von Neumann-based) computers won’t be able to scale efficiently and keep up with the exponential advances in Deep Learning during this decade of the Roaring 2020s.
Being an investor of frontier technologies at Radna Intellectual Ventures, I would like to turn our attention now to Investable Themes in Neuromorphic Computing.
Investable Themes in Neuromorphic Computing
Having talked with VCs on Neuromorphic Computing, I often hear these two complaints.
First, the technology is “too early.”
As Thomas Kuhn noted in his concept of paradigm shifts, technological innovation occurs in jumps. Thus, I believe that extrapolating from prior historical growth of say Digital Computing severely underestimates the adoption rate of disruptive technologies like Neuromorphic Computing.
Second, “hardware is not VC-investable.”
There is a misconception that the only opportunities in Neuromorphic Computing are hardware. Neuromorphic Computing is simply a technology stack like Digital Computing, and as such, there is a Hardware, Software, and an Abstraction Layer connecting the two.
At the Hardware layer, investment opportunities are capital intensive as expected, with the problem compounded by competition from well-funded incumbents like Intel, Qualcomm, and IBM developing their own Neuromorphic chips.
However, opportunities do exist for niche and ultra-low power applications like IoT, with startups like Brainchip competing in this space.
At the Abstraction layer, this is still very much an unresolved problem with limited opportunities until there is a consensus on the abstraction protocol for Neuromorphic computing, similar to the adoption of the x86 instruction set for traditional computing.
That said, Applied Brain Research’s development of the Nengo Neural Engineering Framework is gaining some traction, along with the development of a Neuromorphic Operating System by the EU-funded BrainScales.
Finally, we arrive at the Software layer, where I anticipate most of the investable opportunities for VCs, family offices, and other direct early-investors will lie.
Vicarious and Inivation are two startups working in computer vision and Numenta is licensing their proprietary neuromorphic technology for developers to build upon.
I expect most of these opportunities to be in domain-specific industries where localized Deep Learning is critical like Autonomous Driving, Industrial, and Mobile applications.
Let’s now shift to Quantum Computing, which may be the final frontier of computing.
On the Road to Quantum Computing
I’ll be direct — Quantum Computing in its current state is more hype than reality.
Although IBM, Intel, Google, Alibaba, and Microsoft have all developed “working quantum computers,” I believe these are all pet projects as the fundamental problem of quantum computing – the decoherence of quantum qubits at scale – remains unsolved.
Furthermore, I believe that quantum computing may be the first major technological shift where we need to move away from Silicon as a substrate, and towards organic compounds.
Why is that necessary?
Well, recent research has provided some remarkable evidence that our brain operates as a quantum computer at its most fundamental level.
However, there is limited research and development here, so although I am excited about Quantum Computing, I remain skeptical about its feasibility within our lifetime.
Final Thoughts
Well, if you’re still reading this, hopefully, this 3-Part Series has been informative and can provide you with guidance either as an Entrepreneur or an Investor in Artificial Intelligence.
Returning to our first quote from Part One:
“Artificial Intelligence began with an ancient wish to forge the Gods.”
Indeed, I find it remarkable that we’re on the path to creating a God of some sort — a human-level intelligence devoid of all the fallacies and emotions that ironically make us human.
It is both a humbling experience and an interestingly spiritual one as well.
By Andrew Vo, CFA
Investor at Radna Intellectual Ventures
Related Links: The VC’s Guide to Machine Learning, Creating Music by Machine Learning
About Radna Intellectual Ventures
Radna Intellectual Ventures is a venture startup studio with a mission to enable the next generation of Deep Technologies or Deep Tech companies, with a focus on Machine Learning, Natural Language Processing, and Blockchain technologies.
Our General Partner, Andrew Vo, spent a decade in finance, working at some of the largest investment managers like J.P. Morgan, before pursuing the entrepreneurial path. Andrew serves as an Advisor for startup companies focusing on the FinTech and DeepTech sectors. Andrew is a CFA Charterholder and holds a Masters of Science in Computer Engineering from Cornell University, and a Bachelors of Science in Electrical Engineering and Computer Science from the University of California, Berkeley.
Learn more about Radna Intellectual Ventures here.
— Published on August 17, 2020
