Unsplash

Recurrent Neural Networks, Explained and Visualized from the Ground Up

With an application to machine translation

Recurrent Neural Networks (RNNs) are neural networks that can operate sequentially. Although they’re not as popular as they were even just several years ago, they represent an important development in the progression of deep learning and are a natural extension of feedforward networks.

In this post, we’ll cover the following:

• The step from feedforward to recurrent networks
• Multilayer recurrent networks
• Long short-term memory networks (LSTMs)
• Sequential output (‘text output’)
• Bidirectionality
• Autoregressive generation
• An application to machine translation (a high-level understanding of Google Translate’s 2016 model architecture)

The aim of the post is not only to explain how RNNs work (there are plenty of posts which do that), but to explore their design choices and high-level intuitive logic with the aid of illustrations. I hope this article will provide some unique value not only to your grasp of this particular technical topic but also more generally the flexibility of deep learning design.