Machines and Magic: Teaching Computers to Write Harry Potter

Joyce Xu
Joyce Xu
Jul 3, 2016 · 10 min read

Deep learning for language modeling

I’m going to try to keep these explanations as high-level and non-technical as possible, so anyone already familiar with (or uninterested in) the theory can feel free to just skip this section.

Image for post
Image for post
A vanilla neural net. Source: http://cs231n.github.io,/neural-networks-1/
Image for post
Image for post
Recurrent neural network, unfolded through time (right). XSource: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Word embeddings

Before getting into some of the fun text the Harry Potter language model generated, I want to talk briefly about one of the more interesting parts about the project: word embeddings.

Image for post
Image for post
Word embeddings in the vector space as visualized in a low-dimensional space. Source: http://datascientistdiary.blogspot.com/

The Good Stuff

If you skipped the previous section (or, more likely, started reading it and got bored), here’s where you start reading again.

  1. The original writing contained a lot of stuttering, all-caps yelling, dialogue interruption, etc. that likely threw off a lot of rules the algorithm was trying to learn from the rest of the text.
  2. Preprocessing for the text was not exactly thorough (which was admittedly my fault), and as a result there were quite a few character-level errors that may have also thrown the model off.

So what does it all MEAN?

Image for post
Image for post
Wrong fandom, I know. Source: https://memegenerator.net/instance/38831271
Image for post
Image for post
Enjoyed that read? Click the ❤ below to recommend it to other interested readers!
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Startup Grind

Stories, tips, and learnings from and for startups around…

Joyce Xu

Written by

Joyce Xu

Deep learning, RL, NLP, CV, and all that jazz. @DeepMindAI, @sidewalklabs, @Stanford

Startup Grind

Stories, tips, and learnings from and for startups around the world. Welcoming submissions re: startup education, tech trends, product, design, hiring, growth, investing, and more. Interested in submitting? Visit our submission form here: https://airtable.com/shrShpeN89HrzCzOB

Joyce Xu

Written by

Joyce Xu

Deep learning, RL, NLP, CV, and all that jazz. @DeepMindAI, @sidewalklabs, @Stanford

Startup Grind

Stories, tips, and learnings from and for startups around the world. Welcoming submissions re: startup education, tech trends, product, design, hiring, growth, investing, and more. Interested in submitting? Visit our submission form here: https://airtable.com/shrShpeN89HrzCzOB

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store