NATURAL LANGUAGE PROCESSING
A Concise Description of GPT-3
GPT-3 is a text predictor that can be used to create articles and books, design websites, and do arithmetic
GPT-3 is a new tool, based on a machine-learning algorithm, which is used to predict text. It is released by OpenAI, where Elon Musk and Sam Altman are two of the founders, and both of them have been outspoken about the possibility of artificial general intelligence (AGI), as well as the opportunities and dangers that it may present our society with. Therefore, they wanted to create a company that is developing artificial intelligence in a responsible manner. OpenAI is set out to advance AI, without letting it be able to control us, and they are doing that by creating a friendly artificial intelligence.
GPT-3 is a tool that is used to predict text
Saying that GPT-3 is a text predictor might not sound that impressive. However, the tool is considered a huge step towards AGI and can be used to create believable press releases, articles, books, etc.
Further, it is possible to interact with it, and many people have been creating software where you interact with different personas and ask them questions. GPT-3 has also been used to create strategy documents, design websites, and do arithmetic, and this is only the tip of the iceberg.
The algorithm, which is based on a deep neural network, is explained in great detail in an article that OpenAI published in July. The paper compares GPT-3 towards the state of the art algorithms for specific natural language processing (NLP) tasks. The algorithm is trained only once before it is tested on the NLP tasks, which means that the weights are not being adjusted.
Instead of learning like a traditional machine learning algorithm through thousands of epochs, GPT-3 learns from the input and is able to adapt to other problems, not unlike the way humans learn. Due to the nature of the algorithm and the amount of data (approximately 500 billion tokens) used during training, it has been performing extremely well. The model itself has not been made accessible for developers, who rather can get access to an API.
Developers will, with the API request, provide a couple of examples of what it would like GPT-3 to do, and then this will be used to prime the model and provide the developer with his or her requested text. Priming the model with the best possible examples is important in order to achieve desired results.
Priming the model with good examples is important in order to achieve desired results
GPT-3 has generated a large amount of hype with good reason. However, there are still challenges that have to be solved to make sure it is not being misused. The algorithm is guessing the next word or words by looking at a massive amount of documents. OpenAI has to figure out a way to teach it the social norms we want it to inherit versus what they read in the documents. Hence, a safety net should be included to e.g. filter out racism, sexism, etc.
[1] Frank Chen and Sonal Chokshi. 16 Minutes on News #37: GPT-3, Beyond the Hype (Jul. 2020).
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Amanda Askell, Rewon Child, Prafulla Dhariwal, Sandhini Agarwal, Aditya Ramesh, Arvind Neelakantan, Ariel Herbert-Voss, Daniel M. Ziegler, Pranav Shyam, Gretchen Krueger, Jeffrey Wu, Mateusz Litwin, Girish Sastry, Tom Henighan, Clemens Winter, Scott Gray, Christopher Hesse, Benjamin Chess, Sam McCandlish, Mark Chen, Eric Sigler, Jack Clark, Christopher Berner, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners (Jul. 2020).
