Breaking Moore’s Wall
The Next Epoch of Computing
In 1965, Gordon Moore made a prediction that would set a precedent for technological growth in the modern era. Starting with the invention of the integrated circuit (IC) in 1958, Moore predicted that the number of components in ICs would double every two years. Eventually, his prognosis would be dubbed “Moore’s Law”, a forecast that has been so uncannily accurate it has been widely adopted as the North Star for the semiconductor industry. For over 50 years, companies like Intel would produce next-generation chips on a two-year cycle, multiplying the number of transistors to achieve exponential growth in computing power.
Approaching the Wall
Modern computation owes its foundation to the contributions of Alan Turing and John von Neumann. Turing manifested the study that would come to be known as Computer Science, while Neumann constructed the modern system of computing now known as classical architecture. Today, Moore’s Law continues to guide the semiconductor industry with processors built on the second most abundant element in the Earth’s crust — silicon. (After oxygen which takes up 46.6% of Earth’s mass, silicon is 27.7%.) However, Moore’s prognosis is approaching an end.
First, our appetite for training machine learning (ML) models is dramatically increasing.
“Over the past decade, the demand for AI compute has increased by a factor of nearly 10,000. Ten years ago, the biggest models were 10 million parameters and could be trained in 1 to 2 hours on a single GPU; today the largest models are over 10 trillion parameters and can take up to a year to train across tens of thousands of machines,” — Marcus Gomez (Luminous CEO)
The existing architecture isn’t practical for a world transitioning to net-zero emissions. The Semiconductor Industry Association has stated that we may run out of electricity by 2040 because of our massive computer usage. Or at least, at our current rate, we won’t have enough electricity to meet our computing needs.
Second, as computing, storage, and communication demands continue to escalate, computers based on silicon and conventional architecture run up against physical constraints.
Even though silicon is cheap and extremely abundant, it’s not a perfect conductor — there are other materials with better electronic properties. Silicon has limited mobility of the electrons it carries, meaning that the current moves slower than other materials.
Between the intensified demand for AI and silicon’s limitations, packing more transistors onto a chip won’t always be a viable solution.
As you can tell from the chart above, we are approaching an asymptote in transistor size. The latest transistors in production have been 7 nanometers (nm) long — with the width holding steady at 20–30 nm. Standard ICs are equivalent to ~100 atoms of silicon.
Incumbent chip makers such as TSMC, Intel, and Samsung are pulling out all the stops to launch smaller chips at 5 nm and eventually 3 nm, while IBM claims they will launch 2 nm chips with transistors as tiny as a strand of DNA (~2.5 nm). Since the diameter of a silicon atom is about 0.2 nm, and the wires are typically made with particles of that size, creating 1 nm transistors is highly improbable.
In his second most important paper, Lithography and the Future of Moore’s Law, Moore himself predicted this outcome:
“Capital costs are rising far faster than revenue in the industry. We can no longer make up for the increasing cost by improving yields and equipment utilization. Like the ‘cleverness’ term in device complexity disappeared when there was no more room to be clever, there is little room left in manufacturing efficiency.”
His worries, too, proved prescient — if decades later.
Dennard’s Scaling Law (formulated by Robert H. Dennard & colleagues) states that as the dimensions of a device go down, so does power consumption. However, this law met its end near the beginning of the 21st century because current and voltage couldn’t keep dropping while maintaining the dependability of ICs. As devices become more miniaturized, more complexity in the system is introduced, which creates more bottlenecks in manufacturing. It also results in more latency (i.e. slower computing). Latency has gone up steadily since the Apple 2! (See chart below.)
Physical constraints create performance issues like quantum tunneling, where electrons leap through barriers and cause current leakage. Even though we’ve overcome this barrier in the past, we are closing in on a wall — Moore’s Wall.
Traditional computing has yet to break past these limitations:
- Some algorithms require near infinite memory, but computers today have a finite capacity.
- Computers still can’t think for themselves; humans need to program them.
- There are mathematical proofs of stable and accurate neural networks that exist for a wide variety of problems, that may not have any algorithms that can successfully compute them — even with more data, computing power, or time.
“…there might be a recipe for the cake, but regardless of the mixers you have available, you may not be able to make the desired cake. Moreover, when you try to make the cake with your mixer in the kitchen, you will end up with a completely different cake.” — Anders Hansen (Mathematician at the University of Cambridge)
The unwavering belief in Moore’s Law may have suppressed other technological revolutions — it was only meant to project exponential growth to 1975.
However, it’s not all bleak. After all, in the supreme danger lies the saving power (h/t Friedrich Hölderlin). The industry has been working on taking CPUs out of the driver’s seat — to a post-Moore’s architecture. Just as the famed doctrine fades, a new generation of computing may be on the horizon.
The Coming Epoch
With ongoing geopolitical tensions, the US seeks to gain a strategic advantage and decrease dependence on other countries by onshore chip manufacturing. In the 1990s, the US held 37% of global semiconductor manufacturing capacity, but that has eroded to 12% today. While the federal government plans to subsidize up to $30B in chip manufacturing, chip conglomerates Intel and Samsung plan to spend $117B to build chip factories in Ohio and Texas. The realization of Moore’s Wall is challenging organizations to take things a step further — increasing research and the development of new computing paradigms.
Ray Kurzweil’s Law of Accelerating Returns deduced that technological growth follows an exponential curve, not a constant one. Further, the IC is one of five paradigms (see image below) to guide exponential growth in computing, and as we exhaust one paradigm, we move on to the next. A new epoch is introduced with the next paradigm that can follow in four directions: quantum, neuromorphic, optical, and biological.
Quantum Computing
In the early 20th century, three physicists gave birth to quantum theory — Niels Bohr, Max Planck, and Albert Einstein. Then, Richard Feynman and Yuri Manin proposed the idea of a universal quantum simulator that can harness quantum mechanics’ unique information processing capability.
Quantum vs Classical
Quantum computers use quantum binary digits (qubits) as their basic units of information, and they leverage the unique information processing capability of quantum mechanics. Classical computers are powered by bits, and in a classical system, the bits can only be one state at a time, so 0 or 1, off or on. Theoretically, qubits in a quantum system can exist in a superposition — meaning they can be in multiple states at the same time until measured.
Quantum computers can utilize continuous spin and entanglement properties to function at a much higher level of permutation and combination than classical. If proven viable, quantum computers have orders of magnitude more memory storage and processing power than classical computers.
Taking a Quantum Leap
In the first era of quantum, these machines showed much promise in their expectations to solve complex mathematical, real-world problems that a classical computer could not. Now, we are in the second era, where there is a big push for commercialization. Quantum computing is the fastest emerging market of new computing paradigms — coming from 2017, with $650 million in private capital investments to ~ $3.2 billion by 2021.
Still, the current trend is deceptive because speed comparisons for quantum computers are not always analogous to classical computers. If classical computers are the adult, then quantum technology is an infant. A massive current challenge to quantum proliferation is building hardware capable of mitigating errors caused by decoherence — where disturbances among entangled qubits can cause their computational state to fall apart. To really take the quantum leap we need to go from the noisy intermediate-scale prototypes to fault tolerance while developing early use cases to demonstrate commercial traction.
“We start to see the emergence of alternate qubit modalities, continued consolidation, and an increasing focus on the validity of near term applications.” — Alex Challans
I believe we are just dawning on this industry’s potential, that the second age is a precursor to the disruptive age. As the quantum computing-as-a-service (QCaaS) market grows, developers will be able to push innovation in this space. It will only be a matter of time before enough companies solve enough problems to cement the quantum industry. The need for robust computing that can overcome difficulties involved in drug/material discovery, encryption, etc., will contribute to the growth of the nascent quantum computing industry.
Neuromorphic Systems
Neuromorphic systems are implementations, on a chip, that emulate the neural activities of a brain. So far, attempts at neuronic emulation through standard machine learning models and complementary metal-oxide-semiconductors (CMOSs) have been faulty. Von Neumann’s architecture is unsuited for neuronic emulation because it cannot fabricate memory. Processing neurons require analog chemical signals, yet Von Neumann computers are entirely digital — working with discrete 1s and 0s.
Artificial Computers vs Human Brain
Computers are faster and more precise, but our brains have more storage capacity and nuance in accessing memories, not to mention they’re vastly more energy-efficient,
“In contrast, one of the world’s largest and fastest supercomputers, the K computer in Kobe, Japan, consumes as much as 9.89 megawatts of energy — an amount roughly equivalent to the power usage of 10,000 households. Yet in 2013, even with that much power, it took the machine 40 minutes to simulate just a single second’s worth of 1 percent of human brain activity.”
Classical operating systems lag in training AI models due to their serial method in accessing data files. The solution to achieving Strong AI isn’t putting together more computers with the same architecture. An entirely different kind of multitasking architecture is needed to properly handle the complex behaviors required by cognitive systems. In theory, it makes sense that to simulate true autonomous behavior, you need bulk memory systems that emulate the brain’s ability to process data in parallel.
Brain-on-a-Chip
With processors that imitate the human brain’s neuronal structure and energy efficiency, you can process data in dynamic and unpredictable environments evolving in real-time, meaning:
- Level 5 autonomous vehicles (AVs) and humanoid robots that infer & learn from their environment without being connected to a network.
- Smart sensors in factories processing data around them in real-time to trigger immediate actions.
- Implanted medical early-warning systems adapting to a patient’s state as it changes over time.
- Extremely low-power & always-on detection systems for speech embedded in homes.
- Developing hybrid AI systems, both centralized or decentralized (cloud).
Mimicking the functions of the brain in hardware (such as a neuron and synapse) requires a new set of emerging devices. The most popular approach so far is with a memristor — contraction for “memory resistor”.
Memristors are meant to be simpler than transistors — as they are smaller and could consume less energy. They are classified into six major operation principles that determine electrical behaviors of neuromorphic devices: ionic migration, phase change, spin-based, ferroelectric, intercalation, and ionic gating devices.
Intel has been a leader in this space with its latest, Lohi 2, but there are others pioneering neuronal emulation like Opertran (brain mimicry with insects), GrAI Matter Labs, BrainChip, SynSense, Knownm (building brains on memristors), etc.
Neuromorphic technology isn’t new, but progress has been slow. They require a fundamental change in how hardware and software are developed. For example, “glial cells — the brain’s support cells — don’t figure highly in most neuromorphic designs.” Thus it’s no surprise that chipmakers have been reluctant to invest in the technology until the market has been proven. But since neuromorphic chips can be digital, analog, or mixed, in theory, they can be built flexible enough to complement current computer infrastructures such as CPUs, GPUs, and others — unlocking more avenues for viability.
Optical Computing
Starting from theories of quantum mechanics, scientists Max Born and Emil Wolf were able to derive the Principles of Optics — commencing the revolution to augment light. This field of research has permitted innovations such as mirrors, lasers, fiber optics commonly used to transmit data in communications, and now, computing.
Optical (or photonic) computers send and process information using photons instead of electrons. Rather than an electric current that flows at only about 0.5% of the speed of light (if using Fermi’s velocity, a copper wire has a speed of ~1500 km/s while light moves at ~300,000 km/s) due to resistance — it can be made to perform operations orders of magnitude quicker than can conventional computers. Even in communication, the frequency of large-scale integration (VLSI) computers that use copper wires is capped at 40 GHz, so light may be the only suitable alternative to overcome the limitations of a CMOS.
Accelerating AI with Light
Optical computing is anything but nascent — this field of research dates back 70 years. This time, the critical differentiator for optical computing is that we can do integrated optics with miniaturized devices. Finally, with decades of improvements, the first high-powered computer (HPC) went live in 2021. France’s Jean Zay supercomputer utilizes a photonic coprocessor to speed up randomized algorithms at an enormous scale while working with a standard silicon CPU and NVIDIA’s latest A100 GPU technology.
Startups like Lightmatter, CogniFiber, Luminious, LightOn, and Celestial AI build photonic technologies to scale neural networks faster. Schemes for optical neural networks were first demonstrated in the 1970s but abandoned because silicon offered much smaller and lower power options. Similar to the rest of the computing paradigms, look for photonics to leverage economies of scale due to the waning of Moore’s Law.
Biological Processors
In molecular computing, computation is carried out naturally via biological systems, including cellular materials such as DNA molecules. Since a natural cell can be modified to perform computing without necessarily shutting off the cell’s reproductive capability, a cluster of cells that are too small for a particular computation can grow bigger without human help. This makes a cell equivalent to a computer and a memory fabrication facility. In theory, this would be a parallel computing approach that uses less than 1% of the energy used by current transistors.
Wetware Computation
Since a model of computation is formally defined by inputs and outputs and how an algorithm processes inputs into outputs. The cell can follow the same theoretical model of computation and can be physically implemented in many different ways.
Wetware computing refers to a biological system integrating with traditional silicon computing. Electronic implementations receive data for inputs/outputs, while cells can sense/deliver a wide range of physical, chemical, and biological inputs/outputs. Similar to how information on temperature can be encoded as the height of mercury in a tube, the voltage of an electronic thermometer, or the state of a DNA thermosensor, you can encode biological information. Current designs that explore organic components for biocomputing leverage the information processing units of the cells, such as DNA, gene, or protein circuitries, which are inherently slow (hours to days speed).
Some groups working on bringing biologically inspired computing to life are:
- Cantos portfolio company CATALOG, which is known for storing information in DNA but is also working on enabling massively parallelized compute (e.g. search functions across vast databases) using nucleic acid.
- Researchers from Purdue University found a way to transform structures that occur naturally in cell membranes to create other architectures that are more applicable to computing.
- Nigerian startup Koniku utilizes live neurons from mice to be fabricated onto a silicon chip.
- Roswell Biotechnologies uses single molecules as universal sensor elements to create a programmable biosensor with real-time single-molecule sensitivity and scalability in sensor pixel density.
- Cortical Labs is a young biotech startup that aims to create synthetic biological intelligence (SBI) by teaching 800,000 to 1 million living human brain cells in a petri dish to play Pong. The mini-brains learned how to play the game in five minutes, faster than some AIs.
Although a newer approach to next-generation computing, most of these bio-processors are stunted by age-old technologies, limiting their ability to be low cost and have a moveable device size. They also lack information throughput — the ability to sense and detect a wide range of molecules in one run, though increasing wet lab automation may lower these barriers.
A Heterogenous Future
At first glance, many of these approaches seem outlandish, but in 1961 scientists claimed transistor chips would never be smaller than ten-millionths of a meter — and well, we know how well those presumptions turned out — severely inaccurate, as chips nowadays are 100 times smaller.
Moore’s projection of exponential growth from ten years to forever was the initial technological driver for decades. Although the original projection has reached its limits, new models of computation could be realistic and practical alternatives for driving the information revolution further in ways previously unimaginable.
CPUs won’t be abandoned, but a sea of heterogeneous computational devices will join them. New computing paradigms will alter the heuristics of computing and how to measure technological advancements — we won’t see only one winner, but multiple, each one tailored to its domain and likely working in hybrid systems.
“Moore’s Law has been a glorious journey. While its ending has been more of a soft wane than a dramatic crag, its implications are not. Technology innovation won’t be stopping, but it’s going to be radically different than what the industry has known. Some see it as a call to arms for some creative disruption, others as the freedom to innovate” — Ronni Shendar
