Self-supervised learning is a form of deep learning where a model learns to predict various properties of unlabeled data without explicit supervision. In traditional supervised learning, the machine is trained using labeled examples, while in traditional unsupervised learning, the machine has to learn structure from input data that are not labeled. Self-supervised learning lies somewhere between these two extremes, in that the data is unlabeled, but the model has extra context to use to learn from.
In self-supervised learning, the learning objective is defined by defining an auxiliary task from the input, which can be solved by predicting missing parts. A popular example of this approach is the use of autoencoders. An autoencoder is a neural network that is trained to predict its input given a reduced representation of that input. By training the machine to predict the missing parts of the input, the machine is able to learn useful feature representations in an unsupervised manner.
Another example of self-supervised learning is the use of contrastive learning, where the model learns to contrast the similarity between similar and dissimilar inputs. A popular application of this approach is SimCLR, a technique that learns useful feature representations by comparing the similarities of two augmentations of the same image. SimCLR presents the network with a pair of augmented images, and the task is to determine whether they are similar or dissimilar.
One of the main advantages of self-supervised learning is that it can leverage the abundance of unlabeled data to improve model performance. For example, in computer vision, it is easy to obtain vast quantities of images; by using self-supervised learning, the model can take full advantage of this data, improving its ability to generalize to new, unseen data.
In summary, self-supervised learning is a powerful technique for training deep learning models, which lies somewhere between supervised and unsupervised learning. By defining an auxiliary task from the input, the model can learn to make useful feature representations in an unsupervised fashion. Although still in its early stages in many respects, self-supervised learning represents a promising direction for the future of deep learning.