Creating a simple RNN from scratch with TensorFlow

And using it to build a language model for news headlines

In this article I’m going to explain first a little theory about Recurrent Neural Networks (RNNs) for those who are new to them, then I’ll show the implementation that I did using TensorFlow. We’re going to see the code snippet by snippet along with the explanations and the output that it produced.

The dataset used is A Million News Headlines.

A little theory about RNNs

Let’s first recall what feed-forward neural networks are: they are functions that map the input x to an output ŷ which is an estimate for the true label y. They can be represented like this:

Or, can be drawn like this:

But, the main idea is that they can do only a one-to-one mapping. That is, each input element x should produce only one output ŷ. But what if we need many-to-many, one-to-many, or many-to-one mappings? What…