AI writing stories based on Pictures

Isabella Grandic
Jun 23 · 10 min read

What the heck is an LSTM!

Image result for RNN
#input: 
x = 'CHAPTE'
**Using LSTM: **#predicition:
Y = 'R'
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
print(model.summary())
_________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= lstm_2 (LSTM)                (None, 256)               264192     _________________________________________________________________ dropout_2 (Dropout)          (None, 256)               0          _________________________________________________________________ dense_2 (Dense)              (None, 52)                13364      ================================================================= Total params: 277,556 Trainable params: 277,556 Non-trainable params: 0 _________________________________________________________________ None

Back to my project.

Green boxes are inputs(x), red boxes are output (y), black boxes are the #meat of the model
But first… meme time 😋

#Captioner

The 9x9 grid is the “flashlight”, it’s looking for a certain pattern in an image. The image is the green pixels. We run the “flashlight” over the image, and compile the fancy math on the convolutional layer (blue)
Image result for max pooling gif
Left = convolutional layers, right = max pooling (simplified) layers.
This is the whole VGG network, where the green layers are convolution (or pattern detecting) and blue layers are max poolings (or simplifications)

Given a certain image (in # form), the captioner needs to come up with a caption.

Preparing Data

{2252123185_487f21e336 : bunch on people are seated in stadium}{3457129268_451b88v679 : boy with straw hat} 
input(x)                                                output(y)
startseq Boy
Startseq Boy with
Startseq Boy with lollipop
Startseq Boy with lollipop in
Startseq Boy with lollipop in field
Startseq Boy with lollipop in field endseq
The output of the network is a decoded version of an image.

TL;DR captioner

Captioner’s caption: person reads and sits window
Captioner’s caption: dog on ball … almost there 😅
Captioner’s caption: girl looking away with sunglasses
Captioner’s caption: Person with mask water blue

#Generator

chars = sorted(list(set(raw_text)))
char_to_int = dict((c, i) for i, c in enumerate(chars))
int_to_char = dict((i, c) for i, c in enumerate(chars)
translation is POWERFUL 💪
model = Sequential()
model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))
model.add(Dropout(0.2))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')

Just to recap

Captioner’s caption: many people sit on rocks
[many people sit on rocks] with heads in skye and the sister had  head lap, they mummed togeher in, wind tooo. the fat fat boy ran, and the little sab sing ran and run and woos ‘so much wind blow high prety and book was good to rock soo kid ran and book andfad child . people like eat fod oneanother saw travel bird , sit si fun. time wass padrty and good music we have little rock see hiegh way for long time , sky blue, gooder stuff ahead with peep  (something don’t happen don’t now we go head) context important yes i. we hv and lool hon lets go and go to mountan because we run now, from sibbit!

Isabella Grandic

Written by

Hey, I’m Izzy, a 15-year-old interested in impacting billions with emerging tech. I’m super excited about cellular agriculture and machine learning

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade