Self-Supervised Learning in Vision Transformers

What is self-supervised learning and how has it been applied to Vision Transformers?

Anyone who has ever approached the world of machine learning has certainly heard of supervised learning and unsupervised learning. These are in fact two important possible approaches to Machine Learning that have been widely used for years. Only recently, however, has there been an explosion of a new term, Self-Supervised Learning! But let’s get there step by step and look at the various methods one by one, trying to find an analogy with the human brain.

Supervised Learning is like “learning based on labelled examples”. The model is trained using labelled data so they have been carefully labelled in such a way that each example is associated with a particular class. By studying the characteristics of the various examples of each class, the model learns to generalise and will be able to classify even data it has never seen. To apply this approach, well-labelled data are therefore required and are not always available, and the model may develop biases depending on how the labelling was conducted.