Simple RNN vs GRU vs LSTM :- Difference lies in More Flexible control

Saurabh Rathor
Jun 2, 2018 · 3 min read

When we start reading about RNN (Recurrent Neural Net) and its advanced cells, we are introduced with a Memory Unit (in GRU) and then additional Gates (in LSTM).

However, if we analyse the basic architecture of all 3,

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Basic architectures of RNN, GRU and LSTM cells

we can easily see, there is no such Memory Unit present in the cells. But increasing number of Mathematical operations of Input(xt) and previous output (ct-1, ht-1).

Simple RNN :- Here there is simple multiplication of Input (xt) and Previous Output (ht-1). Passed through Tanh activation function. No Gates present.

Gated Recurrent Unit (GRU) :- Here a Update gate is introduced, to decide whether to pass Previous O/P (ht-1)to next Cell (as ht) or not. Forget gate is nothing but additional Mathematical Operations with a new set of Weights (Wt).

Image for post
Image for post

But, Math operations are performed on same inputs (i.e. xt and ht-1).

Long Short Term Memory Unit (LSTM) :- Here 2 more Gates are introduced (Forget and Output) in addition to Update gate of GRU. And again as above, these are additional Mathematical Operations on same inputs (xt and ht-1). So overall, LSTM has introduced 2 Math operations having 2 new sets of Weights.

Image for post
Image for post

Conclusion :-

Considering above explanation, if we represent these Mathematical operations by Taps/Control Nobs.

Image for post
Image for post

We can say that, when we move from RNN to LSTM, we are introducing more & more controlling knobs, which control the flow and mixing of Inputs as per trained Weights. And thus, bringing in more flexibility in controlling the outputs.

So, LSTM gives us the most Control-ability and thus, Better Results. But also comes with more Complexity and Operating Cost.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store