Neural sonnets

What is it about shakespeare? His language is outdated and convoluted, but yet I think more beautiful and expressive any other writer. There’s something magic in there. As an ML guy, I imagine it to be a lower dimensional essence of poetic verse. I’m going to try to clumsily model it using Python on my macbook.

Of all his writing I think the sonnets are most distinctive, so they’ll be my training data.

I’m going to use a two layer deep model based on Gated Recurrent Units from the elegant Keras API over Tensorflow. GRUs are like lstms, faster, and just as good from my experience. Because Shakespeare literally invented words all over the show, I’m going to use a letter by letter generator to see how it treats the language. I could have used words instead.

First the data, lowering and removing digits to reduce the number of words in my vocab.

filename = “the_sonnets.txt”
raw_text = open(filename).read().lower()
raw_text = ‘’.join([i for i in ‘ ‘.join(raw_text.split()) if not i.isdigit()])

Now to index my characters as ints:

chars = sorted(list(set(raw_text)))
char_to_int = dict((c, i) for i, c in enumerate(chars))

With words this vocab would be very large. Now I’m going to work with a sequence of 100 characters:

seq_length = 100
dataX = []
dataY = []
for i in range(0, n_chars — seq_length, 1):
seq_in = raw_text[i:i + seq_length]
seq_out = raw_text[i + seq_length]
dataX.append([char_to_int[char] for char in seq_in])
dataY.append(char_to_int[seq_out])
n_patterns = len(dataX)

Now reshape the data:

X = numpy.reshape(dataX, (n_patterns, seq_length, 1))
X = X / float(n_vocab)

And one hot encode the output variable:

y = np_utils.to_categorical(dataY)

The actual model, with a bit of dropout to stop over fitting:

model = Sequential()
model.add(GRU(256, input_shape=(X.shape[1], X.shape[2]), return_sequences=True))
model.add(Dropout(0.2))
model.add(GRU(256))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation=’softmax’))
model.compile(loss=’categorical_crossentropy’, optimizer=’adam’)
model.fit(X, y, epochs=20, batch_size=64)

And voila, I can now generate sonnets at will:

start = numpy.random.randint(0, len(dataX)-1)
pattern = dataX[start]
# generate characters
result_arr = []
for i in range(200):
x = numpy.reshape(pattern, (1, len(pattern), 1))
x = x / float(n_vocab)
prediction = model.predict(x, verbose=0)
index = numpy.argmax(prediction)
result = int_to_char[index]
seq_in = [int_to_char[value] for value in pattern]
sys.stdout.write(result)
pattern.append(index)
pattern = pattern[1:len(pattern)]

After the first couple of epochs, it’s not looking that great…

thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee thee

But after 20 epochs, behold:

Neural sonnet I

test, that is to my self with heaven’s erese, 
so thou the pertrn of the world most but toin mine own deartes with deauty soinl, 
to thete coppoued form and siee in thee, 
the summer’s sea against

g, then have i tlee with to the sail of south, 
and srearure of the torlng of touth, 
and dorthnn toon the world in more to sresere, 
and she must be the world shat in more so tourh, 
and so the sime doth

the summer shell ie the counqer with daye i dond of youth in ott and dord then botnt to seem, 
the ounure of aeauty’s faie, 
and therefrre have i sook to seee, 
and mike own measu to me food mone to dome

and sous the world will be teee of me, 
then thou thou move hoo me wour self wotld, 
and therefore have i sook to teee to brere,
in ie the wery with dord of move then more then mens, 
then hate i tourh

of thee, the mongs of thee, 
the bay i domd and saee mone with me then tee, 
then ho the leavt the sieh, 
when i home out toid, 
and then my love thee worth in me, 
now move not move then more then thou

Ok, it’s not perfect but…

I really think it has some of the magic and meter of the original verse. And haven’t you ever wanted to say: of thee, the mongs of thee

It’s interesting how we get real words with other, semi-sensible sounding non-existant words too. RNNs are the business.