Language Modelling — Overview

I love FAQs . They help in addressing important questions about a topic & give us a good overview and understanding of the topic . So , let’s start our journey to understand language modelling . I will break this topic in 3 articles.
- Overview
- N-gram Language Model — explanation with code example in Python
- Neural Language Model — explanation with code example in Python
I would also appreciate people asking questions in comments & their feedback on the topic . It will help me to work on my writing & better communicate .
********************************************************
Q 1: What is language modelling & where is it used ?
Language model learns to predict probability of sequence of words.
Let’s learn its application through some common examples. Have you ever experienced below -
Auto search completion suggestions:

or
email completion in g mail.

Language translation — e.g. Google Translate , Speech recognition — e.g. Amazon Alexa & other smart speakers are among the most popular examples of language modelling .
*******************************************************************
Q 2: Why is the probability of sequence of words calculated ?
This is important to understand .
- Predicting next word
Consider below statement
I went to the …
There are many words which can fill in e.g. Market , School , Office . Probability helps in deciding which word fits the best given the context . For sake of example , let’s say we have 0.8 probability for Market , 0.4 for School & 0.5 for office . It implies that the word “Market” fits the best .
- Similar sounding words —
e.g. Speech recognition system decide on correct text based on probabilities of different options .
- Spell Correction —
Consider following statement:
I love Mhine Learning. Here , spelling of machine is incorrect & it will be corrected after getting similar words & their probabilities .
*******************************************************************
Q 3 : What are various techniques to do language modelling .
We can break techniques broadly in 2 parts:
- Statistical — 1. N-grams , 2 . Hidden Markov model
- Neural — 1. Feed Forward Neural Network , 2. Recurrent Neural Networks .
We will understand more about above models in next set of articles .
*************************************************
Q 4: Can you suggest some good articles on the same topic .
Sure but please do not forget to keep reading mine as well :-).
- Language modelling using LSTM’s by the expert Shivam Bansal .
2. RNN Language Modelling with PyTorch — Packed Batching and Tied Weights
******************************************
In summary , language models are everywhere in our life and very important to understand . Please follow me to check next articles in the series soon.
