Applications of Cohesive-Convergence Groups (Part I): How to visualize Latent Manifold without an Embedding Layer
The journey of training a neural network is like navigating a complex landscape filled with peaks, valleys, and intricate roads. The goal of this journey is to find the most efficient path to achieving the highest accuracy on the training set. However, the main challenge is ensuring an optimal state that’s conducive to the generality of the neural network over the potential test set. In this extended exploration, we delve into the fascinating world of cohesive-convergence groups (CCGs) and their emergence due to the generality of neural networks.
This is the first part of a series about CCGs and their applications in the field of deep learning. In this story, we’ll explore how CCGs act as a window into the hidden world of the latent manifold.
What is a Latent Manifold?
High-dimensional data often resides in a lower-dimensional space. Imagine a swarm of bees buzzing around a flower patch. While their individual positions exist in 3D space (x, y, z), the overall swarm might actually occupy a much simpler, 2D curved surface that defines the shape of the flower patch. This lower-dimensional space is the latent manifold.
Neural networks, when processing complex data, can learn to capture this underlying structure. The latent manifold becomes a compressed representation of the data, highlighting the essential relationships and patterns between data points.
Introduction to Cohesive-Convergence Groups
The generalization ability of neural networks hints that optimizing a neural network on a subset will lead to convergence not only for that set but also for the larger set containing it. This looks like when a flock of birds flies in formation, each bird adjusting its path based on the movements of its neighbors. Similarly, CCGs are clusters of data points within a neural network’s training set that exhibit synchronized behavior during the optimization process. These groups, moving in harmony towards the objective function’s minimum, provide a microcosmic view of the network’s learning dynamics.
Evaluate Cohesive Value of a pair of data points
Discovering algorithms of evaluating cohesive value of pairs of data points is stems from the observation that certain subsets of data points, when subjected to the training algorithm, tend to experience simultaneous increases or decreases in their objective function values. This synchronized movement suggests a form of ‘cohesion’ among these points, hinting at underlying patterns that could be harnessed to analyze the structures recovered by a neural network from data.
Cohesive value is evaluated through a sampling algorithm mentioned in Nguyen (2024) (Algorithm 1). Accordingly, cohesive value of a pair of data points is the number of times the objective function value of two data points in the pair increases or decreases during the sampling process. Naive pseudo implementation of Algorithm 1:
Input: A neural network: U, a subset of the training data: D, a number of sampling steps: n
Output: A table of cohesive values of all pairs of data points in D
Initialize a 2D table T with zero values, accordingly, the cell at position (i,j) contains cohesive value of pair of data points D[i], D[j].
Step1: Draw a random single-training-step neural network U’ with U and a batch in D.
Step2: Sample a batch of data points and mark cohesive pairs in the batch.
Step3: Increase cohesive value of cells at the positions corresponding to the marked pairs.
Step4: Repeat from Step 1 until number of sampling steps, n, is reached.
Return the table T
Reconstruct Latent Manifold with CCGs
The idea of reconstructing the latent manifold from CCGs based on projecting cohesive values of pairs of data points into Euclidean space. The detailed steps are as follows:
- Step 1: Train a neural network with the target dataset (e.g., CIFAR-10).
- Step 2: Choose randomly N data points for latent manifold visualization and construct a NxN map (of size 512x512 in this case) of cohesive values of these data points with Algorithm 1.
- Step 3: Use Multidimensional scaling (MDS) to project the NxN map into n-d Euclidean space.
The result of applying these steps to a neural network with resnet18 architecture He et al. (2015) with the target dataset being CIFAR-10 dataset:
Support me
Enjoying my work? Show your support with Buy me a coffee, a simple way for you to encourage me, and I get to enjoy a cup of coffee! If you feel like it, just click the button below 👇
References
Thien An L. Nguyen. Evidence, definitions and algorithms regarding the existence of cohesive-convergence groups in neural network optimization, 2024. URL https://arxiv.org/abs/2403.05610.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015. URL https://api.semanticscholar.org/CorpusID:206594692.
Source code/Demo — GitHub
