How to get started debugging TensorFlow

Daniel Deutsch
Jul 16, 2018 · 6 min read
Photo by Matthew Kane on Unsplash — https://unsplash.com/photos/5brvJbR1Pn8

Table of Contents

What this is about

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — BRIAN W. KERNIGHAN

Debugging in general can be a tedious and challenging task. Nevertheless, you must be comfortable going through the written code and identifying problems. Normally there are many guides, and the process of debugging is often well documented for many languages and frameworks.

  1. Running the computational graph (using a tf.Session).
Source and credit to https://www.tensorflow.org/guide/graphs

The reference code base

As a reference, I will provide my Github repository with the corresponding code here.

  • tf.train.GradientDescentOptimizer for minimizing the loss

The process of debugging

Now for debugging, there are basically 5 (pragmatic) ways to achieve this.

1. Fetch and print values within Session.run

This is probably the fastest and easiest way to get the information you need.

  • any evaluation can be fetched from everywhere
  • it’s necessary to hold the reference to the tensor, which is bad in complex models

2. Use the tf.Print operation

The tf.Print method comes in handy during run-time evaluation when we don’t want to explicitly fetch the code with session.run(). It is an identity op that prints data when evaluating.

  • it has limited configuration and therefore can easily clog the terminal

It is vitally important that you actually use this returned node, because if you don’t, it will be dangling.

In my code, I added a print statement that fetches the values within the session to illustrate how both methods perform differently in execution.

3. Use Tensorboard visualization for monitoring

Before diving into this debugging method, be aware that there is the Tensorboard and the Tensorboard debugger!

with tf.name_scope("variables_scope"):
x = tf.placeholder(tf.float32, shape=[None, 784], name="x_placeholder")
y_ = tf.placeholder(tf.float32, shape=[None, 10], name="y_placeholder")
with tf.name_scope("weights_scope"):
W = tf.Variable(tf.zeros([784, 10]), name="weights_variable")
tf.summary.histogram("weight_histogram", W)
The distributions tab of tensorboard

4. Use the Tensorboard debugger

As the Tensorboard Github repository states:

This dashboard is in its alpha release. Some features are not yet fully functional.

However, it can still be used and provides cool debugging features. Please check out the Github repository to get an adequate overview. Also, see their video to get a deeper understanding. They have done a great job.

  1. Add your session with tf_debug.TensorBoardDebugWrapsperSession
  2. Add to your tensorboard server the debugger_port
The debugger feature of tensorboard in action

5. Use the TensorFlow debugger

The last method, but also very powerful, is the CLI TensorFlow debugger.

Evaluation metrics while debugging with the CLI

Conclusion

As shown, there are many ways to debug a TensorFlow application. Each method has its own strengths and weaknesses. I didn’t mention the Python debugger, because it is not TensorFlow specific, but keep in mind that the simple Python debugger already provides some good insights!

  • check and sanitize input
  • logging
  • assertions
  • proper use of exceptions
  • failing fast -> immediately abort if something is wrong
  • don’t repeat yourself
  • organize your modules and code

freeCodeCamp.org

This is no longer updated. Go to https://freecodecamp.org/news instead

Daniel Deutsch

Written by

Business Law and Machine Learning. Pushing the limits to make the world a better place. Open for Projects of any kind.

freeCodeCamp.org

This is no longer updated. Go to https://freecodecamp.org/news instead

More From Medium

More from freeCodeCamp.org

More from freeCodeCamp.org

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade