Jul 10, 2017 · 1 min read
Actually, we can work with gradients directly in Tensorflow via optimizer’s `compute_gradients` and `apply_gradients methods`. I’ve done gradient clipping this way and it seem to work fine:
grads = tf.gradients(loss, tf.trainable_variables())
grads, _ = tf.clip_by_global_norm(grads, 50) # gradient clipping
grads_and_vars = list(zip(grads, tf.trainable_variables()))
train_op = optimizer.apply_gradients(grads_and_vars)