Everyone know’s the story of The Room. Lisa, his future wife, breaks his heart. The heroic Investment banker swills Vodka Scotch while playing football in a tuxedo in an alley.
I’m in love with this movie and wished that I could experience more of this movie. After working through Jeremy Howard’s Fast.AI course, I decided to put this knowledge to good use. (http://course.fast.ai/lessons/lesson10.html)
Transfer learning is an impressive technique that can allow a language embedding & a neural network to be applied to a new set of documents. In this example, just as with Fast AI lesson 10, a neural network is taken that had previously been trained on a corpus of Wikipedia documents. The source language model is the wikitext103 LM created by Stephen Merity @ Salesforce research.
We can then take this language model and train the network on an alternative data set. In this case, we train the wikitext103 LM on a fan transcribed script of The Room.
The language model actually doesn’t really use a custom head, but this picture is included because it amuses me.
The main challenge with the use of The Room as the training & test sets is that the corpus is incredibly small and the The Room has very few unique words. Therefore it is important to tweak the dropout rates such that the language model does not over train and is still capable of producing novel combinations of sentences.
#High amount of prediction and likely over training
drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*0.7
#Lower amount of over training and more unpredictability
drops = np.array([0.25, 0.1, 0.2, 0.02, 0.15])*2.25
Now for the good stuff: actual predicted text generated from a training set of The Room.
Less Dropout Regularization:
mark : yeah man , you never know . people are very strange these days . i used to know a girl , she had a dozen guys . one of them found out about it , beat her up so bad she ended up in a hospital on guerrero street .
johnny : what are you doing ?
denny : wake up , johnny , please , please ! its not right ! its not right !
peter : alright , whatever , whatever .
claudette : a man like that , with a gun ! my god !
lisa gets up and heads upstairs .
lisa : what kind of money ?
fade to black .
mark : underwear ? what s that ?
mark : how was work today ?
mark : oh , hey lisa .
More Dropout Regularization:
mike : so , uh , come on , open up . i ca nt wait till later . i still love you . goodnight , lisa . you will ! a man like that ! where s my money ?
lisa : i do nt care .
lisa : ( laughing ) stop it !
denny : i do nt understand you , lisa . i love you . i do nt want to talk to it . that s not worth it . why do nt i come up there and be with you ?
claudette exits .
claudette : what s happening ? he s just being a promotion bitch .
she takes off his shirt and a coffee .
johnny : a — ha — ha . a — ha — ha .
mike : no , you do nt know , know , you ve got plenty of time . its gon na get with your head .
Link to the python notebook on Github below: