Generative and discriminative models are two types of models commonly used in machine learning that approach the problem of modeling the underlying distribution of the input features differently.
A generative model aims to model the joint probability distribution P(X, Y) of the input features X and output labels Y, and then uses Bayes’ rule to obtain the conditional probability P(Y|X), which is the probability of the output given the input. In other words, a generative model tries to understand the underlying distribution of the data, and then use it to predict the output labels. Examples of generative models include:
- Naive Bayes classifier, which models the joint probability distribution using the chain rule of probability.
- Gaussian Mixture Models (GMMs), which model data as a mixture of Gaussian distributions.
A discriminative model, on the other hand, directly models the conditional probability of the output given the input P(Y|X). That is, it doesn’t try to model the underlying distribution of the data, but instead focuses on predicting the output labels given the input features. Examples of discriminative models include:
- Logistic regression, which models the log-odds of the output labels given the input features.
- Support vector machines (SVMs), which find a hyperplane that separates the classes in the input feature space.
The choice between generative and discriminative models typically depends on the specific problem and the available data. If enough labeled data is available, discriminative models may perform better since they directly model the output labels given the input features. On the other hand, when the data is limited, generative models are often preferred as they can generate new samples of data and potentially capture complex dependencies within the data.
In summary, the key differences between generative and discriminative models are that generative models model the joint probability distribution of the input and output data and use Bayes’ rule to predict the output given the input, while discriminative models model the conditional distribution of the output given the input and directly predict the output given the input.