How to Learn Tensorflow the Hard Way

Jupyter Notebook Here

When I first started, I was overwhelmed by all the different high-level libraries on top of TensorFlow such as TF-Slim, Keras, TF-learn, and Tensorflow’s Estimator Class. These libraries contained slightly varying but similar syntax making it hard to follow one example to another. Instead of diving head first into these high-level APIs, I found it valuable to first understand Tensorflow’s low-level API. The learning curve is steeper and not as flashy as implementing VGG16 in 50 lines. But, the payoff is worth your time.

Tensorflow becomes much more straight-forward and malleable once you understand a couple key concepts. I will highlight key concepts here, and I hope others jumping into Tensorflow find them useful.

1. Tensorflow is a dataflow programming framework. (here)
2. Tensorflow Graphs are just protobufs.
3. Tensorflow Sessions statically run Tensorflow Graphs.

Tensorflow is a dataflow programming framework.

In Dataflow or Datastream programming, you define and run a computation graph. Each node in the graph represents a function such as addition, multiplication, etc. Each edge represents the inputs/outputs of these functions. Once the computation graph is defined, we can evaluate outputs of interest.

Figure 1: The figure above is an example of a computational graph. Each node {A, B, C, D, E, F} performs some computational operation. {A} will square inputs, {B} will cube inputs, {C} adds one, etc. Each individual node is dependent only on its inputs. The program is created by the way nodes are wired together.

There are a couple pros summarized from Tensorflow’s documentation:

• Clear dependencies. It is easy to exploit parallelism and distribution across multiple devices (CPUs, GPUs, TPUs, Clusters) because Tensorflow node dependencies (input edges) are explicit. A quick glance at our example shows that nodes {B, C} can be run in parallel, but {C, D} or {B,D} cannot.
• Language agnostic compilation. A graph can be defined in any language of choice, and compiled by TensorFlow’s XLA compiler for optimized compute and memory.

What are the cons?

• Static graph at run time. Tensorflow requires a graph to be predefined before it is run, and once it is running we cannot make changes within that session. In a nutshell, a graph is a network of plumbing pipes and starting a session begins the flow of water. Unfortunately, you cannot move pipes around while water is flowing.

As a note, other libraries such as PyTorch were originally designed to allow dynamic graph execution, but Tensorflow is now adding features for dynamic graph execution (Tensor Fold, Eager Execution).

Dataflow just describes an interface, but how does Tensorflow actually implement their dataflow framework?

Figure 2: Tensorflow’s API implementation which follows a dataflow interface.

I’ll loosely go through the above diagram from the top to the bottom of our diagram.

Programmers use a provided library (Python, GoLang, Java, and C++) which implements the Graph API. These libraries help write out a Tensorflow graph in a special format known as protobuf (we’ll come back to what protobufs are, but for now just think of them as XML, HTML, JSON, or any structuring language).

These libraries also implement the Session API which provides an interface to the Tensorflow C++ Runtime. This runtime is where all the heavy lifting happens. The logic behind each computational node is defined here.

The Tensorflow C++ Runtime will ultimately distribute operations to be executed on the underlying hardware (CPU, GPU, TPU)

So, enough theory — next, we will be focusing on the Tensorflow Python library to create Graphs and then run some Sessions.