Deep Learning Language Model for Telugu Corpus using LSTM
Language Modeling is one of the most important parts of modern Natural Language Processing (NLP) which is used for Text Generation. There are boundless number of areas in which it can be applied, like: Machine Translation, Spell Correction Speech Recognition, Summarization, Question Answering, Sentiment analysis etc.
Here, we build a language model using a Long Short Term Memory (LSTM) RNN dealing with a corpus which is based on one of the Indian Languages- Telugu. RNNs use networks with hidden layers of memory to predict the next step using the highest probability. RNNs use backpropagation to use the text we input to learn how to generate the next letters or characters as its output.
The LSTM part of the model allows us to build an RNN model with improved learning of long-term dependencies i.e., better memory which facilitates an improved performance for those words that we will generate.
About the Corpus
The entire corpus that has been used in this post is in the form of a text file. You can create your own corpus from various sources like Wikipedia or news articles, etc.
Load Data
You can see, the corpus contains sentences in Telugu. Now, we tokenize it to obtain the unique words.
Encoding
We now encode the tokenized words.
Declaring a function that returns the encoded list:
Now we take a window size of 4 words and try to predict the next word that occurs in a sentence. With a stepahead of 3, we split the sentences into small arrays containing the encoded values of the 4 input words and the next word to be predicted. We append both of them into two different lists for the entire corpus.
Array of tokenized one-hot vectors:
Arrays of one-hot representations of words in the window and the next word.
Defining input and target
Training
Defining an LSTM based model:
Training the model for 30 epochs:
Testing
Sample sentence:
Predicting next word:
Conclusion:
You can see that we have successfully achieved the task of predicting the next word, given a set of words from a sentence.
The challenge is to obtain a large corpus and train the model on that.
