Basic Recurrent Neural Network Tutorial — 4

Here we’ll build multi-RNN cell using tf.nn.rnn_cell.MultiRNNCell + tf.nn.dynamic_rnn

If you are interested, the code (jupyter notebook and python file) of this post can be found here.

chentinghao/tinghao-tensorflow-rnn-tutorial

credit: https://blogs.biomedcentral.com/on-biology/2015/10/09/aging-brain-editor-discussion/

Build multi RNN cells

Previously we only build the RNN with one hidden layer, however with deeper neural network comes great performance.

Again with our same input sequence of data

X_data = np.array([
# steps   1st     2nd       3rd
        [[1, 2], [7, 8], [13, 14]],  # first batch
        [[3, 4], [9, 10], [15, 16]], # second batch
        [[5, 6], [11, 12], [17, 18]] # third batch
]) # shape: [batch_size, n_steps, n_inputs]

Define our model

# hyperparameters
n_neurons = 8

# parameters
n_steps = X_data.shape[1]
n_inputs = X_data.shape[2]
n_layers = 5 # 5 hidden layers

# rnn model
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])

layers = [tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons) for _ in range(n_layers)]
multi_rnn = tf.nn.rnn_cell.MultiRNNCell(layers)
output, state = tf.nn.dynamic_rnn(multi_rnn, X, dtype=tf.float32)

Note that MultiRNNCell only accepts the list of BasicRNNCell, therefore we will create a list of 5 BasicRNNCell. Here I prefer to use “layers” in the code instead of “cells”, because hidden layers are the exact things that we are talking.

credit: https://www.youtube.com/watch?v=aircAruvnKk

Now, train the model.

# initializer the variables
init = tf.global_variables_initializer()

# train
with tf.Session() as sess:
    sess.run(init)
    feed_dict = {X: X_data}
    output_shape = sess.run(tf.shape(output), feed_dict=feed_dict)
    state_shape = sess.run(tf.shape(state), feed_dict=feed_dict)
    print('output shape [batch_size, n_steps, n_neurons]: ', output_shape)
    print('state shape [n_layers, batch_size, n_neurons]: ' ,state_shape)

The state is different from what we have seen in single hidden layer.

output shape [batch_size, n_steps, n_neurons]:  [3 3 8]
state shape [n_layers, batch_size, n_neurons]:  [5 3 8]

If you like these series, don’t hesitate to give me a clap below!