Introduction to Self-Supervised Learning (SSL).

The dark matter of Intelligence according to Yann LeCun.

Self-Supervised Learning (SSL) is an innovative deep learning paradigm that is not dependent on human labeled datasets in order to learn concepts. This is important, because in supervised learning, labelling datasets comes with several drawbacks, such as limited scalability. Self-supervised learning is therefore a promising solution for making use of large amounts of data, such as images, text data, raw audio signals or EEG brain signals.

Pretext tasks in SSL: Learning concepts.

In SSL, the model tries to learn a concept without human made labels. Instead, the model uses the structure of the data itself. The idea is to change the original input, for example the image of a bird, by masking out some parts [1][2][3] or dividing the image into patches and randomly shuffle them [4], or apply augmentations such as cropping and colour change [5], etc. The model then learns that the changed version belongs to the same concept of the original image. Human made labels such as “Bird” also represent concepts. However, the main difference is, that the SSL model learns a concept for its input where it is difficult for us humans to name this concept. We can teach the model to associate its learned concepts with human made labels, such as “bird” in the downstream task. This is covered at the end of this article.