Basic Recurrent Neural Network Tutorial — 4
Here we’ll build multi-RNN cell using tf.nn.rnn_cell.MultiRNNCell + tf.nn.dynamic_rnn
If you are interested, the code (jupyter notebook and python file) of this post can be found here.
Build multi RNN cells
Previously we only build the RNN with one hidden layer, however with deeper neural network comes great performance.
Again with our same input sequence of data
X_data = np.array([
# steps 1st 2nd 3rd
[[1, 2], [7, 8], [13, 14]], # first batch
[[3, 4], [9, 10], [15, 16]], # second batch
[[5, 6], [11, 12], [17, 18]] # third batch
]) # shape: [batch_size, n_steps, n_inputs]
Define our model
# hyperparameters
n_neurons = 8
# parameters
n_steps = X_data.shape[1]
n_inputs = X_data.shape[2]
n_layers = 5 # 5 hidden layers
# rnn model
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
layers = [tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons) for _ in range(n_layers)]
multi_rnn = tf.nn.rnn_cell.MultiRNNCell(layers)
output, state = tf.nn.dynamic_rnn(multi_rnn, X, dtype=tf.float32)
Note that MultiRNNCell only accepts the list of BasicRNNCell, therefore we will create a list of 5 BasicRNNCell. Here I prefer to use “layers” in the code instead of “cells”, because hidden layers are the exact things that we are talking.
Now, train the model.
# initializer the variables
init = tf.global_variables_initializer()
# train
with tf.Session() as sess:
sess.run(init)
feed_dict = {X: X_data}
output_shape = sess.run(tf.shape(output), feed_dict=feed_dict)
state_shape = sess.run(tf.shape(state), feed_dict=feed_dict)
print('output shape [batch_size, n_steps, n_neurons]: ', output_shape)
print('state shape [n_layers, batch_size, n_neurons]: ' ,state_shape)
The state is different from what we have seen in single hidden layer.
output shape [batch_size, n_steps, n_neurons]: [3 3 8]state shape [n_layers, batch_size, n_neurons]: [5 3 8]
If you like these series, don’t hesitate to give me a clap below!