Generative AI: how do we build from here?
This is Part 2 of a 2-part series explaining in plain language how Generative AI works, how we got there, and what might await. You can find Part 1 here.
As a VC, it’s easy to feel like you’re missing out. Most investment cycles are led by tech investors’ ability to get swept up in the future. And this time is no different. If we take November 2022 as the mainstream ‘big bang’, when OpenAI’s GPT-3 reached the hearts and minds of millions within a week, the subsequent months have been a Cambrian explosion of new product releases, launches, and announcements.
It’s enough to make anyone’s head spin. So this piece is meant to help build a foundation of understanding for what is actually happening, where we might be going, and to help those who are thinking about building or investing in this space.
There is so much noise and confusion that it’s hard to see where we might be going, but I believe that everyone will eventually need to be using GenTech in order to stay relevant. I also believe that those that will pull ahead in this increasingly competitive space are those that go all-in in one or all of the following: data, computational power, or front-end interfaces.
At the moment, it is incredibly hard to build differentiated companies at the application layer, leveraging existing models and infrastructure. Generative tech’s democratising force has had the impact of flooding the market with applications that use similar underlying models, resulting in nearly identical products and tools.
Now with just a few prompts and a basic level knowledge of coding, anyone can build a whole product, market it, and start selling it. We now have hundreds if not thousands of copycat players, for software writing, content creation, video and image editing, design, synthetic voices, generative music and video games tools, presentations, chatbots. etc.
With regards to foundational models, there are now at least a dozen of players rivalling OpenAI, including Anthropic, Meta, with plenty of other models, both private and open source, quickly catching up.
So what can businesses and founders do to compete or play in this space? As a general purpose technology[1], every company will soon integrate generative technologies to run their business, the way the internet or cloud already is. To really differentiate, companies will have to build a competitive advantage in what I call the power-cells of generative tech: data, computational power, and front-end interfaces.
Data, the (old) new gold rush
One of the main power-cells of generative technologies is clearly data. As we saw in Part 1, models have limitations and will often ‘hallucinate’.
While generative models so far have shown great aptitude at passing the bar or doing taxes, they struggle when it comes to domain-specific tasks or data they have not specifically been trained on. Different types of models, used in autonomous vehicles, medical imaging, or fraud detection for example, will always require large sets of specific data to perform at high levels of accuracy.
This means that businesses and players with huge datasets can leverage generative models in disruptive ways. We’re already seeing this with BloombergGPT and Adobe’s Firefly, which have powerfully enhanced their existing offering simply by applying models to their data.
For foundational model players, this means fine-tuning models for specific usecases and verticals. Google recently launched a new framework called PaLM2, an approach that takes smaller, more efficient models that are fine-tuned for specific tasks. This is already being used to power Med-PaLM 2, which can answer questions and summarise insights from a variety of medical texts.
Panning for data
Models are only as good as their training data and human feedback is incredibly important in these early stages. OpenAI is continuously hiring thousands of contractors to help label data that trains its models. Companies like Scale.AI ($603m raised) and V7 ($44m raised), which help customers manage and label data in automated ways, securely train models, and include human-in-the-loop elements, have proven very successful.
But manipulating data securely remains quite expensive and difficult — not everyone has access to secure testing environments. There are also some scary implications here when it comes to sensitive data, such as financial or healthcare data, or texts and emails. As a result, we will likely see continued advancements and innovation within this space, across data labelling, synthetic data generation, data federation (data collaboration without data sharing), and aggregators of domain specific data.
I believe that some of the most disruptive progress will happen once companies leverage both their internal and external datasets, but there is still a lot of work to be done. Players that manage to secure access to proprietary data, enhanced through customer-facing applications, that they leverage in secure ways, will pull ahead.
Computation is a drag
In the long-run players will need to figure out how to make computing more efficient or all the value will go to the best funded players only. Heartcore aptly points out that SaaS companies using generative tech will see their margins significantly decrease from the added costs of training and inferencing, on top of existing cloud costs. The computational efforts required to train models quickly add up: a16z estimates the cost of training GPT-3 from $500k to $4.6m, per single run, not total.
GPUs, which power the computation of AI models much faster than traditional computing chips, are also in short supply. That said, there has been a shift to locally hosted inferencing, aka edge computation, which would push the cost onto the user. Some models can be run directly onto a Mac or a phone and I believe this is a large part as to why the Apple chips have become so powerful (where they stand here is still a mystery but I doubt they’re sitting idle).
Currently, it appears that most of the value is accruing to incumbents or at least very well capitalised foundational players. Players such as OpenAI ($11.3bn raised), Anthropic ($1.2bn raised), both foundational models, can afford to build and train models, powered by partnerships with Microsoft and Google, respectively.
That said, open source models such as Alpaca or Stable Diffusion prove that it is still possible to catch up, with Alpaca being run on a Google phone. Other players such as MosaicML ($64m raised) can run a model for 15x cheaper than OpenAI and can help companies train and deploy large AI models on their own data while retaining ownership of the model.
Users build your product
A good example here is OpenAI. They are leading the way in terms of mass adoption, reaching 1.8bn visits in May 2023. Having productised their tools and opening up access early on, they’re now pulling ahead, with every new user training their model.
OpenAI is making generative AI accessible. This is what is driving their massive success. In the long-term, we can expect them to try to start to capture the entire value-chain and lock users into their ecosystem, the way Apple has. Currently, OpenAI’s GPT-3 and GPT-4 cannot be fine tuned, or retrained on your own data. You can only prompt it in specific ways, engineering ways of asking the models to get the response you want.
Sure, you could build more custom models and flows if you are so inclined, and platforms like Hugging Face enable you to access hundreds of open-source models for free. But OpenAI’s APIs and plugins enable you to build good products without needing to dive too deeply into the technical complexities of large language models. This approach means it is highly likely that “in time, whenever you encounter a text box on a website or in an app, you’ll find yourself dealing with ChatGPT […] courtesy of an API” says The Guardian.
Most players building applications, tools, or models in this space should build in a way that caters to the everyday developer, business, or tech user as they will be driving most of the returns. The idea is that generative technology, as a tool, should be enhancing human-led projects. Low-code, no-code, human-in-the-loop interfaces with powerful UX and UI will drive mass adoption, which will in return generate powerful data that can help drive model quality.
What it means for people
Founders should be conscious that building applications, without building a moat through proprietary datasets, innovative computation methods, or best-in-class interfaces, will be incredibly difficult.
Incumbents also should be wary — Chegg, an edtech company that helps students with homework questions, saw its value drop by half after showing earnings stunted by AI. That said, this competition can be addressed through the above and in fact Chegg has already released its own chatbot.
On a more personal level, what does this mean for the everyday person? The fear that technology will put us out of jobs or be left behind is not new. And the speed at which change is happening does nothing to help reduce this fear.
Yes, generative AI is a foundational technology, which may revolutionise entire industries, from synthetic protein generation to marketing, from chatbots to medical questionnaires.
Sure, there are huge implications with regards to security and privacy — even bad actors will be more productive. And autonomous AI will likely displace some parts of certain jobs, which will require retraining and will reshape certain careers.
But this will take time and in my view, we won’t all be left behind. It’s likely the opposite. AI will make everyone, even the most unskilled, more productive. Research is already showing that the productivity impact of AI is most pronounced for workers in the lowest skill category. The democratising aspect of generative AI will help remove some of the cognitive load and allow us to be more creative and spend less time on menial tasks.
Sam Altman, one of the founders of OpenAI, put it best:
Like with all technological revolutions, I expect there to be significant impact on jobs, but exactly what that impact looks like is very difficult to predict …I think it’s important to understand and think about GPT-4 as a tool […] and it’s a tool that people have a great deal of control over and how they use it. … GPT-4 and other systems like it are good at doing tasks, not jobs.
I believe these new algorithms, despite their impressive capabilities in reasoning and apparent awareness, are just highly capable automatons, great at fulfilling tasks. There will always be room for human intervention, creativity, and oversight.
If you are building in this space or simply would like to chat further, reach out to blanche@whitestarcapital.com
[1] General Purpose technologies is a technology that generates new technologies: these include combustion engines, cars, electricity, computers, semi-conductors, the internet, and finally AI
