ChatGPT and OpenAI have become a household name since the release of general purpose chatbot ChatGPT on November 30th, 2022. The application went viral within the week, with over a million users registering in an astonishing five days. However, after an initial honeymoon period the novelty faded and significant flaws such as input fragility, bias, hallucinations, and limited quantitative and logical reasoning skills were identified. But despite its flaws the chatbot has dominated the news for over a month now and already stimulated to a new wave of venture capital investments in the tech sector. Google made headlines last month when they called a “Code Red” meeting to address their strategy to deal with the emerging threat posed by OpenAI. First impressions of a sentient supercomputer have faded but OpenAI’s product remains transformative. Let’s begin by looking at what it is, and outline some of the challenges it poses for Google, and what it could mean for the rest of us.
ChatGPT is an application built on the InstructGPT, a natural language model developed by OpenAI. OpenAI is an artificial intelligence company co-founded by Sam Altman and Elon Musk in 2015. Elon Musk stepped away from the company in 2018, but in 2019 Microsoft partnered up with the start-up with a one-billion-dollar investment. This is shaping up to be one of the best billion ever spent as OpenAI has scaled up to become a leader in transformative field of artificial intelligence.
The GPT in InstructGPT (and ChatGPT) stand for Generative Pre-trained Transformer which describe the architecture on which the large language model is built. Large language models such as InstructGPT use these transformers to interpret a user’s input text, interpret it, and produce a response. In this manner ChatGPT can produce high quality, zero-shot responses to interact with people in a natural, conversational manner and perform a variety of NLP tasks such as question answering, document summarization, text generation, and translation, among others. The underlying architecture enables these models to also be tailored for other modalities, as OpenAI has demonstrated with Dall·E 2 (text-to-image generator), Whisper (speech recognition), and Point-E (text-to-3D model).
ChatGPT’s wildly successful beta test poses a few main problems for Google:
- ChatGPT’s release in November gives OpenAI a significant first mover advantage. By acquiring a large user base so quickly, OpenAI has acquired a goldmine of feedback data. Fine tuning on feedback data was how Google was able to get their Flan-T5 model with eleven billion parameters to outperform their sixty-two billion parameter PaLM on the MMLU (Massive Multitask Language Understanding) benchmark. It is unclear how much data OpenAI has collected during this beta test, or how much that data will improve their model, but it will be significant. This is one of the major advantages provided by being a first mover in the AI industry as data follows a law of diminishing returns. While more data is usually better, the information gained from each additional piece of data goes down over time — the tenth piece of data has more informative value than the one hundredth, which has more value than the one millionth. These large language models were trained on data in the scale of a tera and petabytes, but feedback data is limited and more resource intensive; this makes feedback data incredibly valuable, especially in the later stages of development. By getting the user feedback, OpenAI is now able to fine-tune their model with their ample collection of user feedback data which will dramatically improve their future models. The more input data and the shorter the fine-tuning period, the faster the product improves, and the flywheel of production spins. The longer Google withholds its projects from production, the more ground they will need to catch up.
- OpenAI researchers are also going to use the beta period to identify loopholes around their alignment parameters, preventing the chatbot from producing harmful content. Alignment is a field of research in AI aimed at keeping the model performing along with the designer’s goals and interests, preventing them from causing harm. It doesn’t matter how good the AI’s predictions are if the model can’t be put into production, a lesson Microsoft learned in 2016 when they had to pull their chatbot Tay after only sixteen hours.
- A third problem for Google is that Open AI’s ChatGPT has demonstrated significant implications for the future of the search engine industry, where ads make 81.3% of Google’s revenue ($257 billion USD in 2021). Currently they dominate the search engine market, but Microsoft Bing has been growing steadily. In 2022 they reported 11.59 billion, up 67% from 2021. The functionality of Microsoft’s search engine has improved dramatically in the last few years as well, surpassing Google on several key metrics. Microsoft has been investing heavily into Bing the last few years and is now rumored working on integrating OpenAI’s natural language models into Office Suite applications. An interface that integrated a sophisticated chatbot with Microsoft Office and Microsoft Bing would be a powerful product that may pull Googles customers to Microsoft. This would be a big problem for Google as Microsoft dominates workspace productivity software with almost 90% of the market. Googles response to ChatGPT and OpenAI may dictate the future of the company forever.
This is not to say that Google’s natural language models are not powerful. Their latest models released in October of 2022 have bounded ahead of OpenAI’s 2020 GPT-3 engine. While OpenAI has been more secretive about the abilities of InstructGPT, Google is forecasted to have large language models nearing human expert level by 2024. The question is to what degree Google left the door open to OpenAI by not deploying AI products to consumers. Has Google fallen victim of the Shirkey Principle and delay AI product development schedules because they would have cannibalized their lucrative ad revenue? Only time will tell, but OpenAI and Microsoft are moving quickly, and alarms bells ringing at Google HQ.
What does that mean to everyone else?
AI has made some incredible advancements in the last few years, fueled by the widespread adoption of the Transformer which was introduced by a team of Google researchers in 2017. This allowed for the rapid acceleration in the field of AI and has fueled the recent string of achievements by various companies including Google, Meta, and OpenAI. The pace AI is advancing is likely to accelerate as it improves and with venture capital firms lining up to invest in the surge of AI start-ups; funding does not likely seem to be factor. While consumer products are still limited, 2022 saw them start to trickle down and 2023 will see this trickle turn into a stream.
The applications will be limited in scope at first, but once the predictions get good enough, there could be some meaningful change in our day to day lives. Let’s break down what artificial intelligence is and the types of solutions it is capable of. For this exercise it may be better to think of the advancements in AI as breakthroughs in the field of statistics. The value added here is cheap, highly accurate predictions. This is valuable because once the prediction is good enough, it lets humans break down the decision-making process — decoupling decisions into probability and judgement. Probability can be thought of as creating data we are missing, while judgement refers to the ability to decide how much something matters (Power and Prediction, p. 144). A division of labour by specialization allows us to delegate the probability task to machine learning models and free the human labour to focus on judgement. In this manner AI will remove the convenient cloud of ambiguity (and often bias) that is enabled by human-level prediction and allow us to scale judgement with increased transparency and accountability.
When hundreds of workers are laid off automatically, it is not because a machine randomly selected them with a magic black box. It is because someone in head office looked at business predictions made from the data they had and made a judgement that was carried out autonomously. In “Power and Prediction: The Disruptive Economics of Artificial Intelligence,” authors Ajay Agrawal, Joshua Gans, and Avi Goldfarb divide AI solutions into three categories (p. 18):
- “AI point solution: A prediction is valuable as a point solution if it improves an existing decision, and that decision can be made independently.”
Currently these are the low hanging fruit, such swapping the methods of making predictions. Utilizing artificial intelligence to predict risk factors is currently used by insurance industries across a variety of industries. Car insurance companies use driving data to estimate the risk of a driver getting into an accident and then price that into the premiums. While this is something currently done, McKinsey summarizes more advanced implementations in their 2021 article Insurance 2030 — The impact of AI on the future of insurance.
“AI application solution: A prediction is valuable as an application solution if it enables a new decision or changes how a decision is made, and that decision can be made independently.”
Application solutions are not currently as widely implemented, but some industries such as the auto industry have begun implementing application solutions. Outfitting cars with the complex systems are an application change because it enables new decisions being made (the car acting without human input), without changing the system in which it is embedded (public transportation).
“AI system solution: A prediction is valuable as a system solution if it improves existing decisions or enables new decisions, but only if changes to how other decisions are made are implemented.”
Systems solutions are those that require a complete overhaul of the system, just as Henry Ford overhauled manufacturing with the moving assembly line. However, it took four decades of innovation to go from lighting streets to reinvent the factory. This is because system change is expensive and risky. It is highly disruptive. But the value that can be gained from the successful implementation of a new system, is transformative, while the risks of not adopting a new system in time can be fatal.
One industry that is ripe for improvement from AI is the healthcare industry. Highly accurate predictions of healthcare incidents like heart attacks would remove the need for expensive and human resource intensive testing. Streamlining the diagnosis process would free doctors and allied healthcare professionals to focus on treatment. This would allow for more specialized care at lower cost, improving patient outcomes and freeing up resources to be allocated elsewhere.
But we are not there yet with the predictions and may not be for several years. Once we are, the proposed changes would create a change in power dynamic that those about to be deposed would push back. This level of change may not come for a long time, but other system level changes won’t be so difficult. As the value gained becomes more obvious, the barriers will slowly fall.
The AI revolution is here. We are watching cars drive autonomously on devices increasingly filled with AI-powered apps. AI models are already able to hear and see better than us, soon enough they will be able to answer questions better than us as well. This is not to say that they are conscious, but that will be the topic of another post. Bloomberg estimates the industry will be worth over 1.57 trillion dollars by 2030, which is a CAGR (Compound Annual Growth Rate) of 36%. For reference, Industry 4.0 — medical wearables, internet of things, smart factories, industrial automation — is forecasted to have a 21% CAGR between 2021–2030. It’s also important to realize that this won’t be a linear increase in value.
AI economics experts Agrawal et. al believe that the industry will have a more ‘J’-shaped growth curve, reflecting the exponential growth as we come out of the between times — the period between the demonstration of a transformative technology and its widespread adoption (Power and Prediction, p. xiii). Artificial intelligence models are already able to hear and see better than us, they are going to be able to answer questions better than us soon enough as well. This is not to say that they are sentient (that will be the topic of another post) but superhuman AI is here now and it’s only going to get better.
In subsequent blogs I will be continuing to explore the latest developments in AI, expanding on the anatomy of decision making, discussing ways that AI can be implemented in business as well as using transformers to build natural language models. I hope you find this as interesting as I do.
Until next time,
Ryan
