Introduction To DataFlow Graphs As Basis of Deep Neural Networks
Introduction To DataFlow Graphs As Basis of Deep Neural Networks
“Deep Neural Nets” — the buzzword that has been around for quite a while now with its applications ranging from Text Classification to Self-Driving cars, Photo Realistic Image Generation, Music Composition, Robotics and many more yet to pop. Interestingly, the matter of fact is that at the heart of all these super intelligent applications, it’s actually just a stream of mathematical derivatives floating under the hood of such seemingly complex systems.
The Purpose of this Blog Post is to...
explain the mathematics of the dataflow graphs as basis of neural networks from scratch.
The Target Audience is...
Anyone with following prerequisite:
- You know how to compute partial derivatives of a function.
And Now the Roadmap….. 🚵
Milestone #1 : Understanding DataFlow Graphs As Basis of Neural Nets
Milestone #2 : Forward Propagation of A DataFlow Graph
Milestone # 3: Back Propagation of A DataFlow Graph
Milestone # 4 : Forward & Back Propagation in A DataFlow Graph with Numerical Values
Milestone # 5 : Get Your Hands Dirty With A Couple of Other Functions
Milestone # 6: And Finally The Concluding Notes…..
TIME TO DIVE IN!
Milestone # 1: Understanding DataFlow Graphs As Basis of Neural Nets
We will begin with what a “DataFlow Graph” is? Well it’s just fancy word for a graph consisting of nodes and edges.
- Each node represents a mathematical unit of computation.
- Edges on the other hand represent inputs/outputs of mathematical units/nodes.
If you still can’t get clear picture of this, just continue reading and you will understand it in next couple of minutes.
Designing A Simple DataFlow Graph
Let’s build a dataflow graph of a quadratic equation
- Figure out how many operation nodes you have — break down entire equation into individual operations as follows:
It turns out that there are 4 operations nodes.
2. Simply put 4 operation nodes and plug in their inputs as follows:
Milestone # 2 : Forward Propagation of A DataFlow Graph
Forward propagation is a two step process.
1. Compute output of each operation node
2. Compute partial derivative of each output with respect to each of its inputs
Here’s what forward propagation would look like for the given data flow graph:
The values above edges represent inputs/outputs to/of computational modes. On the other hand, tag below each edge represents the computed partial derivative of each computational node with respect to each of its inputs. Similarly, in a neural network, forward propagation involves feeding real world dataset into a flow graph, computing output of each computational node and at the same time caching partial derivatives to be used later in backward propagation. For the data flow graph above, that’s it for forward propagation at the moment.
Milestone # 3 : Backward Propagation of A DataFlow Graph
Backward propagation simply consists of traversing all possible paths from output node to each input node. While doing so, we simply multiply the partial derivatives that we encounter along each path. This indeed is chain rule in action as demonstrated below:
The red circled areas in the above graph represent all the partial derivatives that we computed earlier. All possible traversal paths have been marked in yellow. So let’s say we are interested in computing derivative of final output with respect to node C. This implies that we will simply be multiplying all partial derivatives(chain rule) in the path that exists between node C and the output.
You may be wondering why did we feed an additional “1” that lies below the final output? That is not necessary but for coding purposes, its just a convention to initially feed 1 into the network as a starting point for back propagation as multiplying any value with 1 doesn’t change its value. So no rocket science-just for the sake of coding!
And now, let’s have a deeper look for derivatives with respect to other input nodes as well.
Wondering why should know back propagation when XYZ frameworks will automatically compute it for you?
Here’s what I have got for you to read by Andrej Karpathy( Director of AI at Tesla, Research Scientist at OpenAI and PhD student at Stanford)
Notice that for derivative of input node X, we have added the result of two back-propagated pathways to obtains its final derivative. So this also implies that for instance if a pathway existed before input node X, this “added” derivative’s single value would have been then further back propagated along whatever path that existed.
For the sake of verification, we can cross check derivative formulas computed above using dataflow graph just like we straight away derive them in calculus.
Milestone # 4 : Forward & Back Propagation in A DataFlow Graph with Numerical Values
We would now be feeding numerical values to the dataflow graph just as you would feed a dataset into neural network (to be covered in detail in upcoming blog post).
Assume that we have values as 3,2,1 and 4 for input nodes x,a,b and c. Simply compute values for outputs(squares above edges) and partial derivatives(squares below edges) as shown below:
Derivative of “+” operation is always 1 — Evident in the above flow graph as well.
Derivative of “X” operation is always a switch between its inputs — Refer to partial derivatives of the 2 multiplication nodes in the above graph.
Milestone # 5 : Get Your Hands Dirty With A Couple of Other Functions
This marks the end of dataflow graphs and these are what form the basis of neural network. A deep neural net too is a data flow graph consisting of layers and neurons where each neuron itself is a computational unit of a mathematical function(to be covered in detail in upcoming blog posts). Forward and backward propagation of flow graph enables neural nets to learn patterns from your datasets to make human surpassing intelligent predictions!
Practice Challenge : Design dataflow graphs and execute forward and backward propagation for following functions:
- Sigmoid Function — Sigmoid(x) = 1/(1+exp(-x))
- Quadratic Formula
Remember that to verify authenticity of your dataflow graph, you can cross check the derivative being derived from chain rule in the graph with mathematically deriving out partial derivatives (as we discussed here)
Milestone # 6: And Finally The Concluding Notes….
You now know what goes under the hood of neural networks and those fancy deep learning frameworks.
