Member-only story
A Quiet Shift in the NLP Ecosystem
It is already in motion…
The gist
A paper published a few days ago (15 June 2022) offers a key insight into the impressive task-solving capabilities of the recent spate of billion+ parameter language models.
Large language models exihibit the ability to solve specific tasks when two conditions are met — the model parameter size exceeds 100 billion and the amount of training, exceeds 10²³ FLOPS. Below these thresholds, the capability to solve these specific tasks is for all practical purposes nonexistent.
While another paper from earlier this year confirmed what we have been observing in large language model performance — i.e. language modeling performance improves as we increase model size, data size, and compute budget, this more recent finding from 15th June that model size (and…