Energy-based models (EBMs) are probabilistic models that define a probability distribution over a set of inputs based on an energy function that assigns a scalar value to every input. The lower the energy of an input, the higher its probability. An energy-based model learns a representation of the input data by adjusting the energy function to assign lower energy to inputs that are considered similar or likely according to some criterion.
One common type of EBM is the Restricted Boltzmann Machine (RBM), which is a generative model consisting of a visible layer and a hidden layer. The energy function of an RBM is defined as the negative sum of the product of the weights and activations of the visible and hidden units, plus a bias term for each unit. The RBM learns the joint distribution of the dataset by adjusting the weights and biases to minimize the difference between the expected energy of the model and the empirical energy of the observed data.
EBMs can be used to learn representations for various data types, such as images, text, and speech, in an unsupervised or self-supervised manner. In unsupervised learning, the model is trained on a dataset without any explicit labels, and the representation is learned based on the underlying structure of the data. For example, an RBM can be trained on a dataset of images to learn a set of features that capture the patterns and correlations in the image pixels. These learned features can be used as a starting point for a supervised learning task, such as classification, where the model is trained to predict the label of an image based on its learned features.
In self-supervised learning, the model is trained on a pretext task that is designed to indirectly capture some aspect of the desired representation. For example, an RBM can be trained to predict the missing pixel values in an image that has been randomly masked. By learning to fill in the missing pixels, the model implicitly learns a representation of the underlying structure of the image. This learned representation can then be used for downstream tasks such as classification, object detection or segmentation.
In summary, energy-based models, such as RBMs, offer a flexible framework for learning representations of input data in an unsupervised or self-supervised manner. These representations can be used as a basis for downstream tasks such as classification and detection, and can potentially yield better results than hand-crafted features.