Generating Text with Character-Based Deep RNNs
Literature and the arts are beautiful and fun to look at. To most people, matrices and tensors are not beautiful and fun to look at. With all the computing power we have, we are able to process numbers at an incredibly fast speed, but computers inherently suck at making beautiful and fun things.
Inspired by Andrej Karpathy’s blog, I trained numerous character-based recurrent neural network models to create literature and art for ten minutes each.
Generating Characters
Text is a form of sequential data, and like other projects involving sequential data, sequential models, such as RNNs, LSTMs, and GRUs can be used.
Since the texts we are using contains uppercase letters, punctuation, numbers, and symbols, a word-based model would not be able to access the original text, assuming we do preprocessing: removing punctuation, tokenization, stemming or lemmatization, and removing stop words. This means information is lost, and as a result, the model cannot produce text like the original.
A character-based model would not struggle with this issue, since each punctuation, number, and symbol could be treated as a separate character. In fact, the model would learn the formatting and punctuation patterns of the original text.
For my model, I used a character-based model.
Character Embeddings
If you have worked with natural language processing, you have likely come across the idea of word embeddings, the process of converting a word to a multi-dimensional vector. The embedding (multi-dimensional vector) captures ideas such as:
- The context words appear in
- Analogies between pairs of words
- The frequency of certain words
- Relationships between similar words
Instead of a one-hot vector, word embeddings give the model a meaningful vector to perform calculations upon. Character embeddings perform a similar function, and turn a one-hot vector into something the model can efficiently work with.
There are a few pretrained character embeddings online, but I chose to train my own.
The Model
The machine learning model I used consisted of the following layers:
- Embedding: Learns embeddings for characters, and gives the next layers relevant information.
- Stacked GRUs: Learns when to remember information, and when to forget information. Produces outputs for each input time step.
- Regular Neural Network Layer: Takes the output from the last GRU and assigns a probability to each character with the softmax activation function.
To create the inputs and outputs, I concatenated all my training examples into a single string, and split that string into many sequences of equal length. One training example would consist of a single sequence as input, and a shifted sequence as the target.
Results
I trained models on Shakespeare, news articles, poetry, the book of Psalms from the Bible, piano music from a number of composers, and anime music. Here is a sample of the results:
- Shakespeare
PETRUCHIO:
Nay, that thou then attorn and pale as shepherd's daughter.BIONDELLO:
Where is that you have a study to me.BIONDELLO:
Nay, if they saw some comfort! for thy comfort fortune
It shall be so, being a stranger in this state,
With shame under her good?TRANIO:
What thou camest, of what is death!
O place?
- News
The Chinese of the story has been donating the shooting that he called a pay on what seems to be abortions to be tough of 3-2 a.m. Sunday that the changes in the Australia, In the state abortion regions of neighborhood upon access rain among a mind can send a sense of the unreal nature. Disasters marched discover the next decade because you didn’t want to do that.
"He was trying to questions in the research of the abortion decision to determine whether they felt the finding day of losses was correct."- Poetry (overfitted due to a small dataset):
When A perten scal
While I wrote her letters in bold
Two times four times twelve times two
Night seemed difficult
The world fall in the wirlow
Doing inside my peristy story,
Life gives us bricks to make walking.- Psalms
Psalm 132
A song of ascents.
1 Praise the Lord, all you servants of the Lord
who minst his anointed,
to those who speak evildoers.6 The Lord has done great things for us,
to mad and the Lord heard him;
he hears their cry;
45 for their delight in the Lord.5 May the Lord bless you from Zion,
he who is the Maker of heaven and earth.
The text generated seems to have some sort of structure, and follows some basic grammatical rules, while the music is an absolute failure.
Although the models still have a lot of areas of improvement, the progress from random characters to basic structure and patterns in ten minutes is astounding.
RNNs have exploded in popularity and effectiveness today, and are revolutionizing text generation, translation, speech recognition, summarization, and so many more topics. Their impact on today’s technology is remarkable, and RNNs control the future of artificial intelligence.
Summary
- Computers can make beautiful and fun things.
- Character-based models can learn the patterns in a text and produce similar texts.
- Character embeddings, similar to word embeddings, capture the context and usage of a particular character.
- RNNs have numerous exciting applications in areas like writing and music.
