The Two Seismic Tech Shifts That Will Change Our World
The world is rapidly moving towards two seismic shifts in science and technology that will disrupt economies, change entire industries and create new ones. Either one of these will be sufficient to reshape the global tech economy. Together, their impact can only be imagined through the lens of science fiction. Artificial general intelligence (AGI) and quantum computing will have an impact of such a magnitude that it can be described as the next leap in the evolution of human civilization.
In this article, we look at the current state of each, the principal challenges that need to be overcome, the potential solution pathways and, as we do every year, we highlight key startups on the rise.
The Holy Grail of AI: Artificial General Intelligence
The Rise of the LLM: 2023 was the year of LLMs and generative AI. Ferret, Bard, ChatGPT, You, Llama, and Midjourney, to name a few, are doing amazing things. Generative AI is perhaps the biggest disruptor of productivity since the Internet. LLMs are getting smarter by the day and developing capabilities across modalities giving rise to Large Multimodel Models (LMMs). Unfortunately, they are not perfect and have shortcomings such as explainability, safety, and hallucination (both intrinsic and extrinsic). Much work is being done to address these shortcomings and will continue into 2024. For example, startups such as Gleen AI are addressing hallucination by adding an abstraction layer that sifts through an enterprise’s data to improve the quality of the response.
In addition to the above shortcomings, LLMs can be very expensive. Let’s take a closer look.
The heavy cost of LLMs: A massive challenge associated with foundational general purpose LLMs is the immense compute and associated resources required for both training and inference. With over 1.5 billion visits per month, ChatGPT is said to cost OpenAI in excess of $700,000 per day just to keep it running. It runs on a Microsoft supercomputer that was announced in 2020 and composed of 285,000 processors an 10,000 GPUs at the time. University of Washington researchers estimate the energy required by ChatGPT to respond to all queries for a single day would be enough to power 33,000 households on average [1]. According to a University of Massachusetts Amherst study, a single training run can emit as much carbon as five cars in their lifetimes [2]. This should not be acceptable. In addition, some LLMs make poor use of the GPU resources they run on, with estimates around 30% or lower utilization because of memory and networking bottlenecks. This low utilization rate is yet another example of LLM inefficiency.
The solution: LLMs utilize the concept of attention (transformer model) which scales quadratically with context sequence length. Consequently, they are not sufficiently efficient and struggle to capture long sequences of context. There is a solution emerging that might dethrone the six-year-old attention model. The selective State Space Model (SSM) provides a way to focus on the most relevant information within a long sequence. Originating in the 1960s, the SSM method has been employed to solve a wide range of problems in a variety of applications including time series analysis in economics and finance, control systems, and signal processing. Stanford graduate students developed a new language modeling architecture called Hungry Hungry Hippos (H3) that employs SSMs [3]. H3 allows for much longer context sequences without scaling quadratically [SSMs scale O(N logN) instead of O(N2)]. Researchers from CMU and Together AI have also introduced a new SSM based model called Mamba [4]. Mamba-3B is outperforming Transformer-based models across various modalities, providing as much as 5X higher throughput. SSMs are a great example of pushing the frontier forward by borrowing technology from one discipline into another.
Another solution is to find ways to more efficiently map an ML model onto GPU hardware so as to significantly increase the utilization rate. CentML, a startup out of Toronto, developed a platform for reducing compute while maintaining model accuracy. The result is more efficient utility of computing resources.
GPUs are devouring the data center: The battlefield for sourcing GPUs is intensifying. Better, more efficient models are part of the answer, but it is equally important to address the problem at the hardware level. A wave of AI accelerator startups has surged on the scene, providing training and/or inference solutions at various power budgets. Unfortunately, this is a very complex space to play in because it requires a hardware/software full ecosystem solution that provides for seamless migration. There is certainly room for competition, but AI accelerators that provide only a slight processing edge over Nvidia’s dominant GPU ecosystem may never see the light of day. The difference must be truly significant at scale. Cerebras (focused on training) and Groq (focused on inference) are among the leading contenders.
Is AGI around the corner? Not yet…Machine learning has had tremendous success in extracting correlations, identifying patterns, and making predictions based on historic behavior. However, AGI requires human level understanding, reasoning, and adaptation across diverse domains. An AGI machine would be able to perform general problem-solving, common-sense reasoning, open-ended learning and interacting with the physical world.
The human brain is a marvel of energy efficiency, performing an estimated exaflop (a billion-billion operations per second) while using only 20 Watts of power. The human mind imagines, abstracts, creates reasons, loves, has intuition, dreams, adapts, has an ego and is self-aware. Assertions that AI has now reached the early stages of AGI are premature. There are, however, potential pathways. Let’s look at some of the technologies that may contribute to the push towards AGI:
Causal AI: LLMs base their response on identifying relationships and patterns. They don’t yet have human level causal reasoning. In his book, “The Book of Why”, Judea Pearl, a Turing Award winner, discusses different levels of causal reasoning for AI. The so-called Ladder of Causation is composed of three rungs of sophistication in understanding and thinking: association, intervention, and counterfactuals. Intervention requires causal relationships and counterfactuals require imagining and reasoning about hypotheticals and “what if” questions. Causal AI science might very well be the vehicle to move AI from the narrow towards the general. There are a few startups developing true Causal AI. CausaLens, based in London, is developing a decision-making platform that implements causal model discovery that unlocks cause and effect relationships within the data.
Cognitive Computing (neuromorphic computing): Seeks to learn from the most efficient and most complex machine in the known universe, the human brain. Neuromorphic chips have been around for decades but advances in semiconductor technology are continually enabling more powerful neuromorphic chips. A leading startup, BrainChip, develops event-based processing and spiking neural networks for inference and learning. Innatera, a Delft university spinoff, is developing an ultra-low power Spiking Neural Processor for pattern recognition at the edge. As we learn more about the magic of the human brain, this field of R&D will continue to develop and will inch us closer to AGI.
Brain Computer Interfaces (BCI): In a previous article [5], we reported extensively on BCIs. Invasive interfaces (requiring surgery) are already being used by startups such as Precision Neuroscience, Paradromics and Neuralink to read brain signals. They are also capable of stimulating the brain. As the technology matures, it is likely that BCIs will help unlock some aspects of the human brain that have eluded scientists thus far. Better understanding of the brain will drive technologies such as cognitive computing and will further advance AI towards AGI.
Quantum AI: Bringing together quantum and machine learning could perhaps hold the greatest promise in advancing towards AGI. From quantum neural networks to quantum-inspired machine learning, this marriage of two massively disruptive technologies could bring about the kind of capabilities required to address the world’s most complex challenges. This is an emerging field of research that is attracting considerable attention. Startups integrating quantum principles into machine learning include QuantumMind, and QuantumBrain. Terra Quantum is developing a two phase approach: first run quantum simulations on classical computers and in the future extend to large-scale quantum computers. Startups Aqemia and Menton are improving the efficiency and accuracy of drug discovery by combining AI and quantum technologies.
Quantum Computing: Breaking the Shackles of Digital Bits
The digital world uses bits which can take on one of two values at a point in time. The quantum world dances to the tunes of qubits which can take on multiple values at the same time. Qubits are the foundational computing elements of a quantum computer. Modern digital processors have tens of billions of bits. One hundred error-free qubits, however, can correctly process more calculations at one time than the total number of atoms in the observable universe. Quantum computing will be especially helpful in solving classes of problems that have eluded the capabilities of digital computing:
Complex simulations of exponential complexity such as the dynamic/adaptive behavior of a cancer tumor, leading to the design of more effective drugs.
Revolutionizing the design of new materials: stronger alloys, exotic superconductors.
Optimization problems requiring the navigation of very large multidimensional search spaces, leading to new solutions in just about every sector of the economy.
New levels of cybersecurity with unbreakable quantum encryption protocols
With quantum computing, we will be able to better understand and develop more effective responses to climate change. Quantum computing may even help us unlock the mysteries of the human brain and accelerate the arrival of AGI. For several years now, companies like IonQ and IBM have been providing access to early versions of quantum computers of limited but real ability through existing cloud services.
Achieving the full promise of quantum computing requires solving very complex challenges. Qubits are very fragile and prone to error. Depending on the technology used to create the qubits, it can take hundreds or thousands of physical qubits to be equivalent to a single error free qubit (logic qubit). We need better and more qubits! Unfortunately, as the qubit orchestra expands, keeping all in tune becomes exponentially more complex. Solution pathways include:
Exploring new technologies that provide more stable qubits (longer coherence times) allowing for longer more practical compute times.
New methods for controlling and observing the qubits, allowing for more reliable operation of quantum circuits at scale.
Novel error correcting technology to enable the development of fully fault tolerant quantum computers.
Quality is the key: The quality of the qubit can be much more important than the number of qubits in a quantum computer. Qubit coherence (length of time a qubit can maintain its state), quantum gate fidelity (a measure of the quality of the gate operation), error correction capability and controllability (ability to entangle qubits and form quantum gates) are four of the key parameters. Quantum scientists often refer to the ratio of physical to logical qubits (error corrected qubits) as a key measure of quality and scalability.
Quantum breakthroughs of 2023: One of the most notable developments was the unveiling of Photonic, a startup that is exploiting naturally occurring vacancies in silicon to develop qubits. These unique structures called T-centers have properties ideal for quantum information processing. Photonic researchers have logged remarkable qubit coherence times. T-centers emit light at wavelengths consistent with telecommunications infrastructure, allowing for integration with existing fiber optic networks and paving the way for quantum communications and networking at scale.
Other notable developments in 2023: Microsoft and Intel made significant progress with surface code, a powerful error protocol for fault-tolerant quantum computing, and IBM achieved quantum advantage using its 133-qubit quantum processor Heron on a specific scientific problem. IonQ announced the results of early research for modeling human cognition on quantum circuits. This was a narrow application of cognition but an important milestone that can potentially lead to emulating human-decision processes.
Quantum Dawn: Today, we can actually see the light at the end of the tunnel — quantum computing at scale is within reach! McKinsey estimates a total of 350 quantum technology startups globally (as of 2022). Leading startups include: ColdQuanta and PsiQuantum (USA), Pasqal and Quandela (France), Qutech (Netherlands), Photonic and Xanadu (Canada), OQC and Orca (UK), SpinQ and Origin Quantum (China).
Various approaches are being explored for developing qubits, including superconducting circuits operating at milli degrees Kelvin, atoms and electron spin, trapped-ions and photons. It is not yet clear which of these approaches will get there first. However, technologies that employ existing silicon chip semiconductor manufacturing technology will have a considerable advantage in terms of scaling.
The Next 5–7 Years: AI Evolution and Quantum Impact
ChatGPT 3 was launched roughly five years after the development of the Transformer model. It has been no less than a paradigm shift. The next big shake-up in technology will not take as long. Quantum computing and AI will come together, accelerating the road towards AGI and helping solve some of the world’s most difficult challenges in climate tech, next generation materials, medicine, and the economy. Here are a few predictions through the end of this decade:
Artificial General Intelligence
Prior to 2022, not many anticipated something like ChatGPT coming onto the scene, but the road to AGI is much more difficult to navigate. We are not likely to see human level AGI in this time frame, but AI will continue to evolve at a great pace.
· Today’s LLMs will be a distant past, replaced by much more efficient, high-quality models that can do level 2 reasoning on the Pearl Ladder — a major step towards AGI. Look for a plethora of intelligent AI copilots to choose from that can operate across different modalities and applications.
· In addition to GPUs playing a more dominant role, the data center will see the emergence of additional hardware accelerator chips specialized for different AI applications.
· Intelligence with real-time learning (not just inference) will migrate to the edge.
· Robotaxis like Waymo and Cruise go mainstream at scale in moderate-weather cities across the US.
· There will be growing concerns on safety, security, ethical considerations, and bias leading to significant regulation which hopefully will not stifle innovation and progress.
Quantum Computing
No longer reserved for academic R&D, quantum computing is now providing practical solutions to problems in industry and government. In the late 1970s, classical computing began a transition from the lab and the corporation to small businesses and the home. Access to quantum computing through cloud services will continue to grow, but we will not see a transition similar to classical computing in the next 5 years. Quantum computing will find solutions to currently unsolvable problems, but it is important to keep in mind that it will never replace, nor is it meant to replace, classical computing.
· Physical to logical qubit ratio will decline significantly, paving the way to general quantum supremacy.
· Hybrid classical-quantum computing systems that combine the strength of both classic and quantum computing will play a greater role in solving problems across different sectors of the economy.
· Quantum computing will be a standing component of pharma’s drug discovery platforms. It will help unlock currently unknown disease pathogenesis and usher in a new class of drugs for undruggable disease targets.
The convergence of AGI and quantum computing will bring about unprecedented breakthroughs and help solve some of the world’s biggest challenges. It will be a completely different world.
References
[1] https://www.washington.edu/news/2023/07/27/how-much-energy-does-chatgpt-use/
[2] Energy and Policy Considerations for Deep Learning in NLP, https://arxiv.org/abs/1906.02243
[3] Hungry Hungry Hippos: Towards Language Modeling with State Space Models https://arxiv.org/abs/2212.14052
[4] Mamba: Linear-Time Sequence Modeling with Selective State Spaces, https://arxiv.org/abs/2312.00752
Disclosures: Mubadala Capital Ventures is an investor in Precision Neuro, IonQ, Photonic, and Waymo.
