Why We Invested in Groq
Written by Nicolas Sauvage, President, and Elias Chavarria, Investment Analyst at TDK Ventures
Our decision to invest in Groq in 2020 was driven by our unwavering confidence in the potential of the AI computing market, its environmental implications, and the demand for high-performance, energy-efficient solutions. This foresight has proven to be a game-changer, with some outcomes aligning with our expectations and others surpassing them. In this article, we delve into the rationale behind our initial investment, the aspects that have validated our beliefs, and the aspects that have exceeded our expectations.
Flashback to 2020
Let’s go back to 2020 and explore our beliefs from that time.
TDK’s long-term strategy, with its remarkable foresight, had already identified 5G, AI, and renewable energy as critical innovations for the following ten years. The belief is that AI will evolve across recognition, motion, and context understanding.
Figure 1: AI Evolution [1]
In truth, one of the most powerful capabilities lies in how AI can drive other technologies. Any industry driven by (or that can be driven by) “big data” analysis can be, will be, or already is being revolutionized by using AI to rapidly analyze and process data to produce meaningful, informative output. The gravitas of this capability positions AI to be perhaps the most promising tech enabler of our age — however, it brings a suite of unique challenges to overcome.
2020 — Data as a Problem… and an Opportunity
In 2020, we live in an ever-increasingly digital world — and almost universally for the better, given the associated improvements in quality of life due to digitization. Alongside this digital transformation has been a correlated, meteoritic rise in data generation. Almost 2.5 million terabytes (TB) of data are generated daily, a rate that’s increasing yearly [1, 2]. For context, the Apollo 11 mission moon landing — in terms of computational memory — had 32.8 KB of RAM [4]. That’s 0.000000000001% of the data generated daily worldwide in 2020.
Figure 2. Annual datasphere per year, projected into 2025 [3].
Naturally, the processing of so much data is not free and requires an excessive amount of cost by way of energy. Data centers use an estimated 200 terawatt-hours (TWh) each year, or about 1% of global electricity demand, contributing 0.3% to overall carbon emissions. This metric is similarly rising with data generation [5].
At the core of processing this information is the modern engineering marvel of semiconductors and the resultant integrated circuits that serve as computer processors [7]. These “brains” behind our computers and their ability to process data quickly enable the incredible calculations humanity has already been able to perform. The Apple M1 processor, released in 2020, can perform just over 2.6x109 FLOPs (or 2.6 Tera- floating point operations per second) [8].
Such performance is astounding in itself. However, AI and ML applications bring datasets that outpace even that capability by orders of magnitude. GPT-3, a large language model released in 2020 by OpenAI, has 175 billion parameters. Two challenges emerge — processing such data in a timely manner (the criteria for timely changing with application) and coping with the energy costs associated with that processing. That energy need is also not just a demand for generating more — but a keen reminder of needing to generate the right kind of energy, as the carbon footprint for energy generation represents hundreds of thousands of pounds of CO2.
Figure 3. Energy and carbon impact of data [10][11].
2020 — A Global Market Demand for Compute
With the impetus of all of this context, a global demand becomes clear — in concept, a compute ability to perform, process, and keep apace with the next generation of data epitomized by AI/ML and associated applications, and importantly, a pathway to do so cost-effectively and efficiently, a characteristic that is both important for customer adoption, operation, and environmentally sustainable considerations. More than a concept, however, such a demand calls for compute processing hardware optimized explicitly for AI/ML applications with the innovative performance leaps and novel sustainability solutions necessary to do so effectively.
This driver is reflected by, calculated, and projected in market demands worldwide. A McKinsey Report, “AI Wars: Return of the Hardware,” summarized that the AI market overall is (unsurprisingly) exploding; however, instead of just software alone, it is hardware that will capture about half of the total value. Semiconductors are projected to be one of these, with $60 billion in revenue expected by 2025 from AI-related motivations alone, a massive 20% of the total $300 billion market [9].
2020 — Groq — A Rare Combination of Excellence
As is the case in many avenues of tech development, to improve a system, it is often made more complicated. For compute processors, this can be seen in the hardware changes needed as more silicon chips are added to a processor. Standard practice is for each chip to be divided into space for logic and operations hardware — which defines actual performance potential — while a significant portion (which varies depending on the chip) is taken by caches, controls, and planning functional cores used in managing hardware operation.
Groq rejects this paradigm outright and opts to develop a simplified architecture with a software-first mindset. All hardware associated with caches, control, and planning is removed, freeing up valuable space for more performance. In turn, all control and operations functions are managed by software. This improves performance and efficiency (more compute power per area).
Figure 4. Concept overview of the Groq processing architecture with respect to standard chip design.
Taking things a step further, Groq also changes its approach to memory. A significant issue with respect to AI and big data applications is the necessity to store mass amounts of data — all of which takes time, or in proper terminology, processing latency. In contrast, the Groq processor uses distributed on-chip memory that allows data to be stored and accessed quickly and efficiently; this memory is global and logically shared.
Together, these aspects make the Groq solution — their Language Processing Unit (LPU) — uniquely suited for AI applications from the ground up due to improved performance, efficiency, distributed memory management, and easy scalability.
2020 — A World Class Team
The “magic” that has defined Groq and its innovation is only possible through its one-of-a-kind team — a who’s who of the semiconductor world. Headquartered in Mountain View, CA, they coalesce talent from Google, Intel, AWS, Qualcomm, and others. CEO and Founder Jonathan Ross created the Google TPU.
2020 — Why We Invested in Groq
So, why did TDK Ventures invest in Groq in 2020?
IBM, Intel, and Qualcomm have all enjoyed fantastic growth, but there is a cap on how many computers and smartphones one would ever use daily. Training AI models grows with use cases and improvement iterations. In contrast, AI inference compute grows with usage, and there is no upper limit to the usage of AI models and, by extension, to their environmental impact. We also identified that the total cost of ownership (TCO) drives the purchase decisions of big customers. At the same time, low and deterministic latency is critical for applications such as autonomous ADAS and high-frequency re-training and fine-tuning.
So, it became very apparent the need for effective processing ability to meet the rising demand of AI and associated big data applications — not just in terms of performance but total cost, power efficiency, and latency. While not a technical metric, perhaps the most important is that this need had to be met not just with a concept but with a true working technological solution that proved effective. By all these metrics, Groq distinguished themselves to us (and the rest of the world) as the true King of the Hill of the space.
At that time, the incredible Groq team had already delivered an actual working silicon hardware solution based on their simpler and superior architecture. This solution brought easier deployment with AI developers and order-of-magnitude higher performance than the next best company on IPS, latency (“plus or minus zero” deterministic latency), and total cost. They had garnered significant traction in the industry and proven the scalability of their solution.
In addition to impressive technical prowess and business acumen, the Groq solution also represented significant synergies with TDK Global, which provided the opportunity for substantial collaboration. With the TDK “7 Seas”, Groq AI computing could serve as that pervasive enabler technology that connects with TDK’s interests in wearable tech, AR/VR, data storage, autonomous driving, and many more. There was a natural overlap of interest, which motivated us to share as much “TDK Goodness” as possible to help accelerate them to market.
Figure 5. Groq synergies with TDK corporate interests.
Of course, underlying all of this calculus was a mutually shared vision. TDK Ventures is dedicated to supporting energy and digital transformation for a better tomorrow — and so is Groq. They had contributed significantly to both through their LPU solution, and we believed it would change the game.
Fast forward to 2024
Since we first invested in Groq in 2020, a lot has happened. As we predicted, AI applications have become a market of unlimited scale due to compute requirements not being limited by the number of users. However, there were catalysts for this growth that we didn’t foresee.
In November 2022, OpenAI released ChatGPT, based on GPT-3.5 with its 175 billion parameters. ChatGPT showed the world the impressive capabilities of large language models (LLMs) and generative AI and, potentially even more importantly, lowered the barrier for end users to engage with these technologies. ChatGPT is reported to have reached 1 million users in its first week after launch, and it was estimated to have reached 100 million monthly active users by January 2023, becoming the fastest-growing consumer application in history. By March 2023, ChatGPT 4 was announced, with an estimated 1.76 trillion parameters, more than ten times the number of parameters used by GPT-3.5. This meant potentially higher performance and higher computing requirements for training and running it.
In February 2023, Meta’s release of their Large Language Model Meta AI (LLaMA), with up to 65 billion parameters, marked another inflection point. LLaMA democratized access to cutting-edge natural language processing technology as an open-source AI model, enabling a wide range of users, from individual developers to large corporations, to leverage its capabilities for various applications. This accessibility spurred innovation and competition in the AI field. In a similar timeframe, other players made their product announcements. Google announced Bard (Feb 2023), Anthropic released Claude (March 2023), and Baidu announced Ernie 4.0 (Oct 2023), to name a few.
As all these developments happened, Groq has demonstrated its ability to quickly adapt while delivering a sustainable performance advantage over the competition. Three days after Meta released LLaMA, the Groq team downloaded the model and ran it over a Groq server within a few days. Such an endeavor would have taken weeks to months for much larger teams. Groq’s delivery speed speaks to the team’s adaptability and the efficiency of Groq’s software solution, which is capable of scheduling into any Groq HW system size while delivering top performance. As shown in Fig. 7, in early 2024, Groq’s LPU Inference engine outperforms all eight GPU-based participants across several key performance indicators, including Latency vs. Throughput.
Figure 6: LLM benchmark results by Artificial Analysis.ai [12]
Scaling as an AI cloud services provider
Computing has become the new oil with this new age of generative AI. Forecasts indicate the Generative AI market will grow at more than 40% CAGR over the next ten years and that spending on GenAI solutions will reach $143B by 2027 [13] [14]. For users who want to access AI inference capabilities, acquiring the hardware is one option, albeit expensive. A simpler, faster, and likely lower-cost option is to consume them as a service. Groq has fully embraced this second approach.
Groq has shifted from selling hardware to providing AI cloud services. Its customer is now the AI developer. GroqCloud was launched on March 1st, 2024. As of April 2024, over 80,000 new developers were using GroqCloud, and more than 22,000 new applications were running on the LPU Inference Engine via Groq API. This adoption of GroqCloud indicates a demand for real-time inference at low latency and higher throughput.
We believe that, as demand grows, Groq will be capable of scaling out, partly due to its robust supply chain. For example, Groq does not use high-bandwidth memory (HBM), which is experiencing limited availability for both 2024 and 2025. In contrast, competing solutions, including Nvidia GPUs, rely on HBMs. This all means that Groq can achieve a level of scale nobody else can.
In summary, businesses and consumers are embracing AI-driven solutions and integrating them into various industry sectors. While the use cases today are exciting, we strongly believe there are still many more use cases to be discovered, which will only further grow the AI market and, by extension, Groq’s addressable market. Looking forward, we are confident Groq is well-positioned to continue delivering game-changing AI performance.
References:
[1] TDK Corporation. (2021, May 24) TDK Investors Meeting 2021. Retrieved from https://www.tdk.com/system/files/investors_meeting_2021_en_v2.pdf
[2] Vuleta, B. (2020, Jan) How much data is created every day? 27 Staggering stats, January 2020 https://seedscientific.com/how-much-data-is-created-every-day/
[3] Rydning, D. R. J. G. J., Reinsel, J., & Gantz, J. (2018). The digitization of the world from edge to core. Framingham: International Data Corporation, 16, 1–28.
[4] Kendall, G. (2019, July 1). Would your mobile phone be powerful enough to get you to the moon? Retrieved from https://theconversation.com/would-your-mobile-phone-be-powerful-enough-to-get-you-to-the-moon-115933
[5] Jones, N. (2018). How to stop data centres from gobbling up the world’s electricity. Nature, 561(7722), 163–166.
[6] ‘Tsunami of data’ could consume one fifth of global electricity by 2025. (2017, December 11). Retrieved from https://www.theguardian.com/environment/2017/dec/11/tsunami-of-data-could-consume-fifth-global-electricity-by-2025
[7] Introduction to semiconductors. (n.d.). Retrieved from https://www.amd.com/en/technologies/introduction-to-semiconductors
[8] Sohail, O. (2021, October 19). Apple’s M1 Max teraflops performance is higher than the PS5, 4x faster than M1, according to fresh comparison. Retrieved from https://wccftech.com/m1-max-teraflops-performance-higher-than-ps5/
[9] McKinsey report “AI wars: Return of the hardware”, Feb 2019
[10] Andrae, Anders (2017, October 5). Total Power Consumption Forecast. Retrieved from https://www.researchgate.net/publication/320225452_Total_Consumer_Power_Consumption_Forecast
[11] Hao, K. (2019, June 6). Training a single AI model can emit as much carbon as five cars in their lifetimes. Retrieved from
[12] Groq. ArtificialAnalysis.ai LLM Benchmark Doubles Axis To Fit New Groq LPU™ Inference Engine Performance Results. Retrieved from https://wow.groq.com/artificialanalysis-ai-llm-benchmark-doubles-axis-to-fit-new-groq-lpu-inference-engine-performance-results/
[13] Bloomberg. (2023, June 1) Generative AI to Become a $1.3 Trillion Market by 2032, Research Finds. Retrieved from https://www.bloomberg.com/company/press/generative-ai-to-become-a-1-3-trillion-market-by-2032-research-finds/
[14] IDC. (2023, Oct 16) IDC Forecasts Spending on GenAI Solutions Will Reach $143 Billion in 2027 with a Five-Year Compound Annual Growth Rate of 73.3%. Retrieved from https://www.idc.com/getdoc.jsp?containerId=prUS51310423
