[Tensorflow] Core Concepts and Common Confusions

From a beginner’s point of view (Understanding Tensorflow Part 1)

Ceshine Lee
Mar 29, 2018 · 8 min read
Image for post
Image for post
Photo by Luca Bravo on Unsplash

A Little Bit about Myself

  1. I’ve been using Keras since its pre-1.0 era, and have mostly used it to train some standard MLP/CNN/RNN models.
  2. Last July I started to learn PyTorch, and have since been able to work with some more advanced or customized models using PyTorch.
  3. I had worked through the course and exercises when Google’s udacity deep learning course came out, so I can read a Tensorflow project and get a gist of what’s going on. But I’ve never done any real project directly with Tensorflow before. It’s still very foreign to me.

Why Tensorflow

I’ve been considering picking up Tensorflow for a while, and have finally decided to do it. The main appeal of learning Tensorflow includes (compared to using PyTorch exclusively. Keras as a high-level library is not really comparable.):

  1. Tensorboard is awesome. Granted, there are already Tensorboard integrations for other deep learning libraries. But using it with Tensorflow will be more well-documented and well-supported.
  2. Production readiness. Other libraries are catching up, but Tensorflow has it already all figured out for you(Tensorflow Serving and TensorFlow Lite).
  3. There have some voices asking people to boycott Facebook’s research ecosystem given the recent turmoil at Facebook (the company behind PyTorch). Although I personally still feel conflicted about the idea of boycotting a very fine piece of open source software, learning the other major alternative can do no harm.
Image for post
Image for post
Tensorboad Graph Interface

The Tensorflow Programming Stack

Image for post
Image for post
Source

Graphs and Sessions

This is the one of the most important concept for those who come from PyTorch. Pytorch create a dynamic computational graph on the fly to do automatic differentiation. Tensorflow, on the other hand, requires you to define the graph first. I

Image for post
Image for post
Data Flow in a Graph
# Graph definition
graph = tf.Graph()
with graph.as_default():
# Operations created in this scope will be added to `graph`.
c = tf.constant("Node in graph")
# Session
with tf.Session(graph=graph) as sess:
print(sess.run(c).decode("utf8"))

Session.run

This method is basically the whole point of creating a session. It starts the data flow in the graph (pumps water into a piping system). The two most important parameters are fetches(outputs) and feeds(inputs).

# Define a placeholder that expects a vector of three floating-point values,
# and a computation that depends on it.
x = tf.placeholder(tf.float32, shape=[3])
y = tf.square(x)

with tf.Session() as sess:
# Feeding a value changes the result that is returned when you evaluate `y`.
print(sess.run(y, {x: [1.0, 2.0, 3.0]})) # => "[1.0, 4.0, 9.0]"
print(sess.run(y, {x: [0.0, 0.0, 5.0]})) # => "[0.0, 0.0,25.0]"

Tensors v.s. Variables

In PyTorch, a variable is part of the automatic differentiation module and a wrapper around a tensor. It represents a node in the computation graph, and stores its parent in the graph and optionally its gradients. So basically all tensors in the graph are variables.

saver = tf.train.Saver()with tf.Session() as sess:
sess.run(init_op)
# Do some work with the model.
inc_v1.op.run()
dec_v2.op.run()
# Save the variables to disk.
save_path = saver.save(sess, "/tmp/model.ckpt")

Name Scope v.s. Variable Scope

PyTorch names the parameters in a quite Pythonic way:

Image for post
Image for post
PyTorch Parameter Naming
Image for post
Image for post
The Graph of One of My Models
Image for post
Image for post
Left: tf.variable_scope Right: tf.name_scope (Source)
with tf.variable_scope("foo"):
with tf.name_scope("bar"):
v = tf.get_variable("v", [1])
x = 1.0 + v
assert v.name == "foo/v:0"
assert x.op.name == "foo/bar/add"

Time to Get Our Hands Dirty

That’s pretty much it! In this post we have covered 4 topics that I’ve found most confusing to beginners:

  1. Graphs and sessions
  2. Tensors and variables
  3. Name scope and variable scope

Quick Links

Veritable

Towards human-centered AI. https://veritable.pw

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store