The #paperoftheweek 8 is: “Topology of Learning in Artificial Neural Networks”

Motivated by the lack of understanding of how neural networks learn, the authors, in this very didactic publication, argue that it should be possible to capture this learning evolution through a topological analysis, loosely defined as the mathematical study of qualitative structures and shapes. They represent the evolution of weights, between each hidden layers of a fully connected neural network designed for a classification task, as a Mapper graph that encodes the topological structure of the cloud points from the weights. By interpreting the graph and its branches they show how the weights evolve accordingly with the training iteration steps and how their trajectories define, in some cases, smooth surfaces. The possible extensions of their work include “… studying how learning graphs are impacted by initialization schemes, optimization methods, regularization, biases, activation functions, and depth. It would also be interesting to explore learning graphs for convolutional and recurrent neural networks.”

Abstract:

“Understanding how neural networks learn remains one of the central challenges in machine learning research. From random at the start of training, the weights of a neural network evolve in such a way as to be able to perform a variety of tasks, like classifying images. Here we study the emergence of structure in the weights by applying methods from topological data analysis. We train simple feedforward neural networks on the MNIST dataset and monitor the evolution of the weights. When initialized to zero, the weights follow trajectories that branch off recurrently, thus generating trees that describe the growth of the effective capacity of each layer. When initialized to tiny random values, the weights evolve smoothly along two-dimensional surfaces. We show that natural coordinates on these learning surfaces correspond to important factors of variation.”

You can read the full article here.

About the author:

Ignacio Alvizú Fiedler, Deep Learning Researcher at Brighter AI.