Perplexity is an evaluation metric that measures the quality of language models. In this post, we will discuss what perplexity is and how it is calculated for the popular model GPT2.
What is a Language Model?
You might have already heard of large language models(LLM) such as BERT, GPT2 etc., that have changed the face of Natural Language Processing. There are different types of language models such as Statistical language models and neural language models. The SLMs are based on statistics in the given text where as the Neural language models are trained using neural network architectures.
At its core, a language model(LM) is nothing but a probability distribution over a set of words which is known as vocabulary of the model. It tells the probability of a certain word in the vocabulary occurring given all its previous words. Usually, whichever word that has the maximum probability is selected as the next predicted word in the sequence.
This probability can be calculated by multiplying a sequence of conditional probabilities for each word given its previous words which gives the likelihood of this sequence.
For example, the joint likelihood of the example sentence “It is a beautiful day” is written as shown below. Calculating this probability helps us to predict next or…
