Beyond Language: Introduction to the Transformational Power of LxMs
by Vic Singh and Archit Gadhok
A core tenet of the new firm that I am building is to generate sectoral theses in areas that I believe can lead to outlier returns. With that goal in mind, I am energized by this multi-part series where we delve into the transformational potential of foundation models across various industries. In this post, I introduce the concept of LxMs — a new thesis I am developing — and how they will impact various sectors.
This series will explore how these advanced models are revolutionizing fields such as scientific intelligence (biology, chemistry), industrial intelligence (robotics, manufacturing) and gaming (design, engines, modeling, simulation).
In this post, which will be the first of a multi-part series, we will set the stage by sharing our overarching vision and approach to this fascinating topic. Subsequent posts in this series will provide in-depth insights and case studies on how foundation models are being leveraged within each specific industry.
Early last year, I published an article on Foundational Open ecosystems and followed it up with a three-part series on the Open Source AI stack. In those pieces, we discussed frameworks for companies that are building fundamental layers of technology in an open source way and serve as platforms for broader interconnected ecosystems for other companies and developers. We also discussed how we believe this open-source approach drives long-term innovation across sectors such as bio, compute, data, energy, finance, genomics, healthcare, identity, gaming, infra, IoT, media, music, privacy, robotics, science, social, storage and more.
As I think about what comes next for me, I am energized by novel foundation technology — catalyzed by open ecosystems — creating boundless opportunities across the technical stack for decades to come and this series is meant to investigate the beginnings of just that — the application of foundation models to areas beyond languages.
Introduction
Today we are introducing the concept of LxMs — a term we’ve coined to describe foundation models that extend beyond traditional language applications, and can have diverse & variable applications. These models exploit multi-modality and are tailored for a variety of industries. Unlike typical models trained only on publicly available internet data, LxMs utilize proprietary datasets, which can be transformational for specific sectors — such as biology. LxMs are finding widespread application in areas such as scientific intelligence, industrial intelligence, and gaming, demonstrating their potential to drive advancements and innovation in these fields. Our upcoming blog series will delve into the possibilities and impact of LxMs across various industries.
Foundation models: To quickly introduce the concept, Foundation models are unsupervised AI neural networks that are trained on vast datasets, which enable these foundation models to understand and generate human-like text, recognize images and sounds and even make predictions based on incomplete information.
While earlier machine learning models were trained for specialized tasks on specific datasets, the beauty of foundation models lies in their generalizability. Foundation models primarily learn from unlabeled datasets, and can be easily applied across multiple applications with minimal finetuning.
To date, foundation models created for natural language processing have dominated all the spotlight. OpenAI’s GPT3 and GPT4, Google’s Gemini and Amazon-backed Anthropic’s Claude are all models that have revolutionized text generation, translation, summarization and more. However, the utility of foundation models extends far beyond language, and has potential to revolutionize other industries as well — from life sciences (biology, chemistry), healthcare to industrial manufacturing (robotics) and climate sciences.
A differentiating feature for foundation models is the ‘transformer architecture’ on which they are based. The transformer is a type of neural network architecture that relies solely on an attention mechanism to draw global dependencies between input and output sequences. It was introduced in 2017 and quickly became the backbone for state-of-the-art language models due to its parallelizable nature and ability to model long-range dependencies effectively.
Other than the transformer architecture, foundation models are different (as compared to AI/ machine learning models previously developed), because of the exponentially high number of parameters these models are trained with. For e.g., OpenAI’s latest model GPT 4o has been trained on >175B parameters, which is higher than most other models trained in history. This enhances the model’s learning potential and generalization, versatility across tasks, creativity, scalability and efficiency.
The need for a framework: As the name suggests, LxMs are applicable to any domain where these large-scale models can be applied and this introduces a certain complexity in systematically understanding, categorizing and leveraging the application of these diverse foundation models across industries.
To systematically understand this space, we developed a framework. We call it the Verticals x Capabilities framework. This framework is a tool for mapping the intersection of industry verticals with the modes of data inputs (language, audio, vision) that these foundation models can process.
Verticals: Through this part of the framework, we intend to identify the industries (x in LxMs) where significant work is happening in the development of foundation models. To identify these verticals, we not only conducted secondary research to identify hotbeds of development, we also jammed with a number of experts (investors, operators, researchers and so on) to develop a nuanced perspective on the space.
Capabilities: These span the spectrum of data modalities that current state-of-the-art foundation models can process, including language, vision and audio.
A framework such as this helps us in a structured analysis of the ongoing developments, identification of innovative application areas, and observing cross-industry trends.
Based on industry trends that we are observing and through our conversations with numerous investors and operators in this space, there are 3 areas that are of special interest to us.
LxMs utilize expansive and diverse datasets — from genetic sequences in computational biology, kinematic data in robotics, to player behavior in gaming — to learn and adapt, thereby enhancing precision and functionality in complex environments across multiple sectors. Here is an overview of what is forthcoming in this series:
Scientific intelligence
- LxMs have found widespread application in the sciences, especially in biochemistry. Models are trained on vast quantities of protein (Proteomics) or genome (Genomics) data to solve long standing scientific problems of drug discovery, drug development and therapeutics. While there are a lot of up and coming startups in this space, BigTech giants such as Alphabet have also thrown their weight behind this space. For e.g., Alphabet’s Alphafold 2 model was developed to predict protein structures. Outside of biochemistry, there is some nascent work in fields such as astrophysics, material sciences and so on — and these are areas that we are very excited about.
Industrial intelligence: LxMs are making significant inroads into industrial intelligence, notably within robotics that is creating inflection points across major GDP sectors from manufacturing to logistics.
- Robotics: Within robotics, LxMs are revolutionizing the way robots learn, interpret and interact with their environment. Audio and vision-based foundation models allow robots to better understand and navigate their environment.
- Manufacturing: In manufacturing, LxMs will drive advances in predictive maintenance, quality control, and supply chain optimization. The integration of AI into manufacturing processes fosters a more resilient, flexible, and efficient production environment, paving the way for the next generation of smart factories.
- Logistics: LxMs can be leveraged in several ways to optimize logistics and supply chain operations. Use cases such as demand forecasting, automated inventory tracking to warehouse optimization can be tackled using foundation models.
Gaming: In Gaming, LxMs can be used by both game developers as well as middleware and tools developers — both during production and run-time. Custom models fine-tuned with the lingo and lore of specific games can help understand context, generate human-like text and respond to in-game prompts in a coherent manner. They’re designed to learn patterns & language structures and understand game state changes — evolving and progressing alongside the player in the game. We truly believe that gaming in the paradigm of LxMs will lead to massive breakthroughs and incredible gaming experiences that we are yet to imagine.
Use cases identified within gaming:
- Intelligent agents / Intelligent Non playable characters (NPCs)
- AI-powered animation (e.g., facial animations) — using state of the art diffusion models -
- Asset creation for games — using state of the art diffusion models
- Game testing — intelligent agents to simulate game testing
Conclusion
In summary, the rise of LxMs across the vertical and capability-focused foundation model landscape is rapidly evolving, and we will soon be at a stage where the full potential of foundation models is realized across all major GDP sectors and facets of society.
I am always looking to partner with the boldest and most original founders building domain-specific foundation models, the infrastructure that optimizes them and the tooling that enables the great applications to be built atop, in any vertical. If you’re one of those builders, please reach out thevicsingh at gmail or @vicsingh on Twitter. Regardless, stay tuned for our subsequent posts, where we dive deep into each of these verticals!
