Galactica — Stochastic Parrots in action?

(Almost) real image of Stochastic Parrots generating outputs for large language model :) (generated with Stable Diffusion model).

A demo of Galactica (named after Isaac Asimov’s Encyclopedia Galactica), a large language model (LLM) trained on 48 million scientific articles, was made available last week by Facebook’s parent company Meta. Two days later, the AI community pulled the model amid debate over its propensity to produce inaccurate or misleading content.

In this article, we’ll try to examine the drama that has been unfolding and determine whether Galactica can actually live up to our expectations or if it’s just a stochastic parrot that generates random nonsense while posing as a knowledge portal.

Where can I get some of these Stochastic Parrot cuties?

Unfortunately, I have to disappoint you here… you can’t. Stochastic parrots is a term introduced by the authors of this paper. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? The general idea is that large language models can be compared to parrots — they will mostly repeat the statements seen in the training data (Parrot), slightly tweaking the prediction (Stochastic). Moreover, they can do it better for concepts that are well and frequently described so they can struggle to describe concepts that are underrepresented.

The article describes several issues and limitations of large language models. The most…