A Quiet Shift in the NLP Ecosystem

It is already in motion…

Figure 1. An abstract illustration of emerging and growing capabilities of language models with increased model size. The utility of large language models goes beyond their ability to generate coherent stories — they could be leveraged to solve almost any NLP task — both generative and discriminative. Examples of this are noun phrase tagging, NER, relation extraction, classification, coreference resolution, semantic search (sentence embeddings), etc. Also for some of these tasks, the performance may be adequate for production use cases with fine-tuning. Image by Author — using information from Google’s PaLM model blogpost

The gist

A paper published a few days ago (15 June 2022) offers a key insight into the impressive task-solving capabilities of the recent spate of billion+ parameter language models.

Large language models exihibit the ability to solve specific tasks when two conditions are met — the model parameter size exceeds 100 billion and the amount of training, exceeds 10²³ FLOPS. Below these thresholds, the capability to solve these specific tasks is for all practical purposes nonexistent.

Figure 2. Image from a recent paper (15 June, 2022)

Figure 3. Image from a recent paper (15 June, 2022)

Figure 3a. Image from a recent paper (15 June, 2022)

While another paper from earlier this year confirmed what we have been observing in large language model performance — i.e. language modeling performance improves as we increase model size, data size, and compute budget, this more recent finding from 15th June that model size (and…