Open Language Models

Everything You Should Know About Evaluating Large Language Models

From perplexity to measuring general intelligence

Image generated by the author using Stable Diffusion.

As open source language models become more readily available, getting lost in all the options is easy.

How do we determine their performance and compare them? And how can we confidently say that one model is better than another?

This article provides some answers by presenting training and evaluation metrics, and general and specific benchmarks to have a clear picture of your model’s performance.

If you missed it, take a look at the first article in the Open Language Models series:

A Gentle Introduction to Open Source Large Language Models

Perplexity

Language models define a probability distribution over a vocabulary of words to select the most likely next word in a sequence. Given a text, a language model assigns a probability to each word in the language, and the most likely is selected.