A Fun Way to Tinker with Neural Networks
This article introduces Tensorflow Neural Network Playground (TP) to provide a brief overview of NN for the wonderers on the topic’s tinkering stage. TP is an interactive visualization created by Daniel Smilkov & Shan Carter and written in typescript using d3.js. It contains a NN library that meets the demands of this educational visualization. You can simulate NN in solving given classification, and regression problems see the results, in real-time, in your browser. But first, let’s look over the basic terms of NN within the scope of TP.
Epoch: It is defined as one pass over the entire dataset. So every time the epoch number increases, all training data complete their trip. In the TP interface, the reset button resets iterations and weights to 0. The step button advances one iteration.
Weight: The parameter within a neural network transforms input data within hidden layers. At the first iteration, random weights on synapses between neurons move through the network, known as forward propagation. The work of adjusting this weight using the method of gradient descent is called backpropagation.
Learning Rate: Learning rate is a critical hyperparameter that is used to speed up the procedure to get local optima. During each iteration, the gradient descent method multiplies the learning rate by the gradient. The final output is called the gradient step. Determining the learning rate is the same as deciding how fast weights change for each iteration.
Activation Function: It is for deciding on a simple mathematical operation. And it is responsible for activating the ‘hidden layer neurons in the network. Let’s mind an activation function with hard output. The inputs and weights are multiplied and then summed together. This value is compared to the threshold value. If it is larger than the net-input value, the output is 1. If not, then it is zero. It is called a hard output activation function. Available soft output activation functions in TP are ReLU, Tanh, Sigmoid, and Linear.
Regularization: It is a hyperparameter that is used to prevent overfitting. Available values in TP are L1 and L2. L1 computes the sum of the weights. On the other side, L2 takes the sum of the square of the weights.
Features: It represents the input layers of the network to feed in.
Hidden Layer is the layer between input and output layers, where artificial neurons take in a set of weighted inputs and produce an output through an activation function.
Output: The output layer in the NN often involves the loss evaluation. The loss function is a method of evaluating how well the NN performs in the given data. If predictions deviate too much from actual results, loss function will be high.
Setup 1
Now let’s look into our first setup. The objective of this setup is to conduct a classification of the Gaussian dataset given by TP.
Learning Rate: 0.03
Activation Function: ReLU
Regularization: None (Regularization is not needed to solve a simple problem because overfitting will most likely not occur.)
Features: X1 and X2
Network Structure: 1 hidden layer with 1 neuron
In this setup, only one hidden layer with one neuron is used. After it is started, it is seen that it will end fast successfully. The Test Loss and Training Loss values become very small quickly, and their loss curves (black and grey curves) overlap.
Before, the NN could not distinguish between the orange and blue data points. After training, it is seen that the blues dots are in the blue region and the orange dots are in the orange region, which shows us that the classification process is succeeded.
Setup 2
Let’s try to classify the more complex dataset pattern shown in Figure 2.
Learning Rate: 0.03
Activation Function: ReLU
Regularization: L2
Regularization Rate: 0.003
Features: X1 and X2
Network Structure: 2 hidden layers with 8 neurons and 2 hidden layers with 6 neurons, respectively.
The challenging problem of this setup is the swirled structure of orange and blue data points. It is needed to make NN deeper by adding hidden layers and neurons to succeed in classifying this dataset. The objective of the design should be what is the smallest number of hidden layers and the neurons to succeed in the classification.
In addition to these setups, you can try increasing the number of neurons in the hidden layers, other activation functions, additional cross-product features, or other transformations like sin(X1) and sin(X2) to classify the given problems. You can also use the following tools and references for further thought about NN.
References
[1] Google Cloud Big Data and Machine Learning Blog [Online]. https://cloud.google.com/blog/products/ai-machine-learning/learn-tensorflow-and-deep-learning-without-a-phd
[2] Google Machine Learning Crash Course with TensorFlow API’s [Online].
https://developers.google.com/machine-learning/crash-course/ml-intro
[3] Neural Networks, Manifolds, and Topology [Online]
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology
[4] Tensorflow — Neural Network Playground [Online].
