Language Modelling — Overview

Pulkit Mehta
Nov 7 · 3 min read

I love FAQs . They help in addressing important questions about a topic & give us a good overview and understanding of the topic . So , let’s start our journey to understand language modelling . I will break this topic in 3 articles.

  • Overview
  • N-gram Language Model — explanation with code example in Python
  • Neural Language Model — explanation with code example in Python

I would also appreciate people asking questions in comments & their feedback on the topic . It will help me to work on my writing & better communicate .

********************************************************

Q 1: What is language modelling & where is it used ?

Language model learns to predict probability of sequence of words.

Let’s learn its application through some common examples. Have you ever experienced below -

Auto search completion suggestions:

or

email completion in g mail.

Language translation — e.g. Google Translate , Speech recognition — e.g. Amazon Alexa & other smart speakers are among the most popular examples of language modelling .

*******************************************************************

Q 2: Why is the probability of sequence of words calculated ?

This is important to understand .

  • Predicting next word

Consider below statement

I went to the …

There are many words which can fill in e.g. Market , School , Office . Probability helps in deciding which word fits the best given the context . For sake of example , let’s say we have 0.8 probability for Market , 0.4 for School & 0.5 for office . It implies that the word “Market” fits the best .

  • Similar sounding words —

e.g. Speech recognition system decide on correct text based on probabilities of different options .

  • Spell Correction —

Consider following statement:

I love Mhine Learning. Here , spelling of machine is incorrect & it will be corrected after getting similar words & their probabilities .

*******************************************************************

Q 3 : What are various techniques to do language modelling .

We can break techniques broadly in 2 parts:

  • Statistical — 1. N-grams , 2 . Hidden Markov model
  • Neural — 1. Feed Forward Neural Network , 2. Recurrent Neural Networks .

We will understand more about above models in next set of articles .

*************************************************

Q 4: Can you suggest some good articles on the same topic .

Sure but please do not forget to keep reading mine as well :-).

  1. Language modelling using LSTM’s by the expert Shivam Bansal .

2. RNN Language Modelling with PyTorch — Packed Batching and Tied Weights

3. https://towardsdatascience.com/machine-learning-text-classification-language-modelling-using-fast-ai-b1b334f2872d

******************************************

In summary , language models are everywhere in our life and very important to understand . Please follow me to check next articles in the series soon.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade