PoKi Poems Text Generation — A Comparison of LSTMs, GPT2 and OpenAI GPT3
PoKi is a corpus of 61,330 poems written by children from grades 1 to 12. PoKi is especially useful in studying child language because it comes with information about the age of the child authors (their grade). Github Link
A sample PoKi poem:
A snowy day : one snowy day the children went outside to play in the snow. they threw snowballs, went sledding and made a snowman. afterwards they went inside to drink warm hot chocolate. it was a fun snowy day
We compare text generation methods using the PoKi corpus. We try some of the traditional model approaches like LSTMs and then the newer and more advanced OpenAI models. For our use case, we haven’t differentiated the poems by age, however that can be explored further.
1.Lets start with the classic Andrei Karpathy’s character RNNs. We tokenize the text based on characters and encode it. There are 129 unique characters. We process the text by joining all the titles and the poem text and keep the sequence length for each sample as 100. The model is a simple LSTM model with a character embedding layer and a Dense output.
BATCH_SIZE = 64
vocab_size = len(vocab)
embedding_dim = 256
rnn_units = 1024def build_model(vocab_size, embedding_dim, rnn_units, batch_size):
model = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab_size, embedding_dim,
batch_input_shape=[batch_size, None]),
tf.keras.layers.LSTM(rnn_units,
return_sequences=True,
stateful=True,
recurrent_initializer=’glorot_uniform’),
tf.keras.layers.Dense(vocab_size)
])
return modelmodel = build_model(
vocab_size=len(vocab),
embedding_dim=embedding_dim,
rnn_units=rnn_units,
batch_size=BATCH_SIZE)def loss(labels, logits):
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
We train the model for 20 epochs and get output for a prompt like this
“I Love The Zoo : roses are red”
Output:
i love the zoo : roses are red, violets are blue, does not mean life!
dear good bye : cherries are whispery and you have feet may be cold five yet are not found make an illusions feel the bad gentle featured air, it’s stabalay the bend like a milutor that i can tell the points through things me the dawn to drift, the strikes of agliosable people not knowing why? what kids really want to do without? should i thru. my tears falls, still as a frozen lake.
Hmm, yeah some word twisters there, but pretty good considering it just had the character structure information.
2. Lets look at Word based LSTMs. We keep the model structure same as above, only replacing the vocab size with the word vocabulary size of 60726. The model training takes much longer due to the much larger vocabulary size.
And for the same prompt, here’s the output :
i love the zoo : roses are red,write trains toes and legs drowning as i think of a simlie i feel calm like a catapiller or as free as the wolf who saw it a night but the only thing is to have my chance to the beach even my old snout i wonder if iâ’m not my face feel well for the depth of my consciousness one night i walk here and see in her arms i know she is wrong when she likes
Mind numbing hehe ? :)
3. LSTMs with FastText Embeddings
We will now try with FastText Embeddings. We train Fast Text on the corpus and then use it as an embedding matrix in the LSTM model. We train it for 20 epochs. For the same prompt as above, we again generate the output.
Output:
i love the zoo : roses are red,make a little with once more imagine when i was nine i had to get in the grade up in the hay it was drinking out i saw the honey bee how come i could come up today but yesterday id be you gonna have lots of homework so things quick to make it to make friends things sometimes things can be when a friend can be an orchestra bunch of mother should you agree in poems so little boy can’t get out of school in school but boy heart everybody time i’m dreaming
4.OpenAI GPT2
Lets try with GPT2 before moving on to GPT3.GPT2 large has 774 Million parameters and is almost 3GB in size. Owing to machine constraints,we will use only the small version comprising of just 124 Million parameters
Using HuggingFace, we can very easily import the tokenizer and the model and use it for prediction.
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
GPT2 = TFGPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)Greedy search selects the word with the highest probability at each time step. The model starts repeating itself though.
Output from Greedy Search:
I Love The Zoo : roses are red, violets are blue. i love the zoo. do you? The scary forest. : the forest is really haunted. i believe it to be so. but then we are going camping. A Hike At School : the school is really scary. i believe it to be so. but then we are going camping. A Hike At School : the school is really scary. i believe it to be so. but then we are going camping.
Beam search find the most likely number of possible words at each time step and continuing to expand along that path. It eventually chooses the hypothesis which has the highest probability. Beam search can also output repetitions and we use no_repeat_ngram_size = 2 to remove that confound.
Output from Beam Search:
I Love The Zoo : roses are red, violets are blue. i love the zoo. do you? The scary forest. : the forest is really haunted. i believe it to be so. but then we are going camping. A Hike At School : I am going to go to school. I will be in the middle of the woods with my friends. We will have a good time.
We can use sampling to introduce some randomness in the kind of text we generate. We set do_sample = True. In Top-K sampling, the K most likely next words are filtered and the probability mass is redistributed among only those K next words. This reduces the possibility of generating incoherent words.
Output from Top K sampling:
I Love The Zoo : roses are red, violets are blue. i love the zoo. do you? The scary forest. : the forest is really haunted. i believe it to be so. but then we are going camping. A Hike At School : a hike at school. a hike is a hike. i am not sure if it is a hike or a hike. but i think it is a hike.
Huh. Well.
5. OpenAI GPT3 — Finally the King arrives :) We can access GPT3 using the OpenAI API. Lets see what GPT3 generates.
Lets give two titles along with the poem text for GPT3 to understand what we want.
Prompt :
“I Love The Zoo : roses are red, violets are blue. i love the zoo. do you? The scary forest. : the forest is really haunted. i believe it to be so. but then we are going camping. A Hike At School: i took a hike at school today and this is what i saw bouncing balls girls chatting against the walls kids climbing on monkey bars i even saw some teachers’ cars the wind was blowing my hair in my face i saw a mud puddle, but just a trace all of these things i noticed just now on my little hike. Computer :”
Output of Text completion:
“1 2 3 code-click 4 enter-enter 5 ok-sleep Keyboard : on the keyboard i like my lowercase on the left upper case is on the right numbers live between them i do not like the keys together it makes my mistype my texts i like a space in between. Twins : there are a lot of things that come in pairs they have second eyes to see you know how things look even better how”
Its amazing that by looking at just a few lines and a pattern, GPT3 is able to understand that its a title, followed by the poem text and is able to emulate it with quite human-like sentences.
Lets try some other use cases with GPT3.
English to French translation:
Prompt:
“English: How are you ? French:Quel est votre nom? English: What is your name ? French: Wie lautet dein Name ? English: Where are you from ? French:”
Output:
“D’où venez-vous? English: How old are you ? French: Quel âge avez-vous ? English: What is your address ? French: Où habitez-vous ? English: What is your email address ? French: Quel est votre adresse e-mail ? English: What is your telephone number ? French: Quel est votre numéro de téléphone ? English: Can you speak English ?”
It seems to catch the general pattern quite well !
Summarization:
Prompt:
“By design, a blockchain is resistant to modification of its data. This is because once recorded, the data in any given block cannot be altered retroactively without alteration of all subsequent blocks. For use as a distributed ledger, a blockchain is typically managed by a peer-to-peer network collectively adhering to a protocol for inter-node communication and validating new blocks. Although blockchain records are not unalterable, blockchains may be considered secure by design and exemplify a distributed computing system with high Byzantine fault tolerance. The blockchain has been described as an open, distributed ledger that can record transactions between two parties efficiently and in a verifiable and permanent way. Summarizing this we can say “Output:
“As a distributed ledger, blockchain reduces the costs of verifying transactions, and improves efficiency of transactions across multiple parties. For use as a distributed ledger, a blockchain is typically managed by a peer-to-peer network collectively adhering to a protocol for inter-node communication and validating new blocks.”
That looks quite amazing.
Free form writing:
She told God in the folds of her soft white sweater
Wearing a cadre without the ties
He said, “Little ones, do we leave you behind? Do we permit?”
She felt the warmth of Heaven and sighed once and said, “Goodbye.”
She looked down and said she was doomed
Standing under Heavens door; He opened the door, and said, “Youre not doomed;
One day star, you”Prompt:
“Once there was a monkey and a duck who were good”
Output:
“ friends. One day they decided to go exploring.As they were walking a road late one night, they came to a fork in the road.”Which road do we take?”The duck asked.The Monkey said”Why decide, when you can have a coin toss?” So the Monkey flipped a coin into the air. They walked down the road the coin landed on and so they did not decide!”
Wow !
Question and Answer:
Prompt:
“Me:Hi! Can you help me with some good restaurants ? AI: Sure, what cuisine would you like ? Me: I prefer continental AI:”
Output:
“ let s see, for a beautiful sea view is the Just A fish restaurant Me: That s great! Please book for tomorrow evening we ll be with 4 people AI: Is this Angela and Rodrigo ? Me: Yes! We will go there!”
Its scarily awesome isnt it ? There are just so many things which can be done using GPT3 by showing an example and letting GPT3 infer the appropriate completions.
All the code is available on the github repo.
Hope you liked the article ! Do let me know your comments.
