Memory Leak — #33
VC Astasia Myers’ perspectives on AI, cloud infrastructure, developer tools, open source, and security. Sign up here.
🚀 Products
This week, Google released Gemma, their state-of-the-art open AI born out of the Gemini project. Gemma comes in two model weights: 2B and 7B. Both are pre-trained and can run on a developer laptop or desktop with CPU or GPU — and Google Cloud with GPU and TPU acceleration. The Gemma 7B model outperforms other open models on various benchmarks, such as MMLU and GSM8K. Currently, Gemma only offers text-to-text capabilities.
Why does this matter? Over the past year, open source models like Llama, Alpaca, and Mistral joined the scene as an alternative to OpenAI. Originally, it was unclear if cloud providers would offer open models. Interestingly, Google made a distinction between “open models” and “open source models” by stating, “Open models feature free access to the model weights, but terms of use, redistribution, and variant ownership vary according to a model’s specific terms of use, which may not be based on an open-source license. The Gemma models’ terms of use make them freely available for individual developers, researchers, and commercial users for access and redistribution. In using Gemma models, developers agree to avoid harmful uses, reflecting our commitment to developing AI responsibly while increasing access to this technology.” AI engineers had mixed thoughts on a particular line, “Google may update Gemma from time to time, and you must make reasonable efforts to use the latest version of Gemma.” Builders wondered what constituted “a reasonable effort.” Quickly platforms like Ollama launched support for Gemma.
Enabling the AI Infrastructure on Arm
Arm introduced a new Neoverse Compute Subsystems that promises significant improvements in performance and efficiency for AI workloads. This advancement in Arm’s infrastructure technology is poised to further accelerate the adoption of AI in cloud computing and infrastructure by providing optimized integrated platforms that cater to the evolving demands of hyperscalers, startups, and various industries.
Why does this matter? Nvidia remains the dominant solution for AI chips; however, there are GPU shortages and it is expensive. Other chip providers are looking to serve the market and catch-up. AI builders could be empowered by additional environments to build, train, and run their AI services. This week we also saw Groq demonstrate performance gains because it bypasses two bottlenecks that GPUs and CPUs get stuck on: compute density and memory bandwidth.
GPT users can now rate GPTs and provide private feedback directly to the builder. Additionally, GPTs “About” section can now include: builder social profiles, ratings, categories, # of conversations, conversation starters, and other GPTs by the builder.
Why does this matter? OpenAI continues its efforts to become a modern app store by allowing reviews. We’ve come across startups that have launched GPTs as a growth hack to help accelerate adoption of their core product. We predict that this year at least one startup will make a name for itself from its OpenAI GPT.
📰 Content
Scaling ChatGPT: Five Real-World Engineering Challenges
Gergely Orosz of “The Pragmatic Engineer” interviews Evan Morikawa, who led the OpenAI Applied Engineering team as ChatGPT launched and scaled. Evan discusses the five engineering challenges along with lessons learned. Challenges are: 1) KV Cache & GPU RAM, 2) optimizing batch size, 3) finding the right metrics to measure, 4) finding GPUs wherever they are, and 5) the inability to autoscale.
Why does this matter? ChatGPT launched in November 2022 and grew to 100M WAU in November 2023, unprecedented growth. ChatGPT’s model uses the transformer architecture, a key characteristic of which is that each token is aware of every other token. This means it scales quadratically. In turn, scaling out the service was particularly hard not just because of the astronomical user growth, but also the inherent model architecture.
Nvidia’s Stock Market Value Rose $273 Billion in a Day
Nvidia’s stock price jumped 16% on Thursday, increasing the company’s market value by a staggering $273 billion in just one day, a record amount. The chipmaker has been seeing soaring demand for its semiconductors, which are used to power AI applications. The company’s revenue more than tripled in the latest quarter compared with the same period a year earlier.
Why does this matter? Nvidia, which has positioned itself as one of the most prominent players in AI, has been producing some eye-popping numbers. Nvidia’s ride is breathtaking. It has surged 67% this year — adding nearly $710 billion in value. Nvidia is the third-most valuable S&P 500 company at ~$2 trillion. The company continues to be the main provider of AI chips, leading to increased venture capital investment in alternative form factors.
Knock gives developers a better way to build notifications. Their APIs and component libraries help engineers ship product notifications that engage users and drive long-term retention. Today products like Webflow, Vercel, Amplitude, and hundreds of others trust Knock to power notifications in production. They announced a $12M Series A led by Craft Ventures.
Why does this matter? Often engineers are responsible for building notifications. Sometimes every new feature shipped means new notifications to build across the frontend clients and out-of-app channels. This can take weeks and features often shipped without notification support, leading to poor engagement and retention. Knock solves this problem by offering a notification infrastructure platform that enables teams to move faster. It is also an example of marketing solutions that targets developers like Segment.
💼 Jobs
⭐️DragonflyDB — React Tech Lead — Dragonfly Cloud
⭐️Chroma — Member of Technical Staff
⭐️Speakeasy — Product Engineer
Views expressed in posts and other content linked on this website or posted to social media and other platforms are my own and are not the views of Felicis Ventures Management Company, LLC. The posts do not constitute investment, legal, tax, or other advice and do not constitute an offer to invest in any security.
