Bayesian Machine Learning involves making probabilistic inferences based on data. Computing the posterior distribution over the model parameters is crucial for this task. However, computing the posterior distribution is often intractable in complex models, which makes it hard to do inference using traditional methods. To overcome this, variational inference offers a way to approximate the posterior distribution with a simpler, tractable distribution.
Variational inference is a form of optimization that allows the approximation of the posterior by minimizing a distance measure between the true posterior and the simpler tractable distribution. This measure is typically the Kullback-Leibler (KL) divergence, which quantifies the distance between the two distributions.
More formally, let’s consider a Bayesian model where we wish to infer the parameter vector θ given the data 𝒟. The goal is to compute the posterior distribution p(θ ∣ 𝒟), which can be written using Bayes’ law as:
$$p(\theta\mid\mathcal{D})=\frac{p(\mathcal{D}\mid\theta)p(\theta)}{p(\mathcal{D})}$$
where p(𝒟 ∣ θ) is the likelihood function, p(θ) is the prior distribution over the parameters, and p(𝒟) is the marginal likelihood or evidence.
Using variational inference, we approximate the true posterior p(θ ∣ 𝒟) with a simpler distribution q(θ), which is chosen from a family of tractable distributions, such as Gaussian or Gamma distributions. We choose q(θ) as a member of this family, then optimize its parameters to minimize the KL divergence between q(θ) and the true posterior p(θ ∣ 𝒟):
q*(θ) = arg minq(θ)KL(q(θ) ∥ p(θ ∣ 𝒟))
The optimal q*(θ) is the one that is the closest to the true posterior distribution under the KL divergence measure. The computation of the KL divergence involves computing the integral of q(θ) and p(𝒟 ∣ θ)p(θ), and approximately computing it either by Monte Carlo methods or by analytic approximations when possible.
Once we have found the optimal q*(θ), we can use it to perform Bayesian inference. For instance, we can compute the mean or mode of the distribution, or we can compute credible intervals (similar to confidence intervals) to make probabilistic predictions on new data.
Variational inference has become popular in recent years because it allows fast and scalable inference in complex models. However, it does not guarantee that the approximate posterior distribution is a good approximation to the true posterior, especially in high-dimensional spaces or when the true posterior is highly irregular. Therefore, it is important to check the quality of the approximation by computing convergence diagnostics, such as the evidence lower bound (ELBO), to ensure that the approximation is valid. Additionally, the choice of the family of distributions for q(θ) can impact the accuracy of the results. For example, a Gaussian distribution may not be a good approximation if the posterior is bimodal or if it has long tails. Therefore, the choice of the family of distributions needs to be carefully considered based on the features of the posterior distribution.
In conclusion, variational inference is a powerful tool in Bayesian Machine Learning that allows the approximation of the posterior distribution with a simpler, tractable distribution. This approximation allows faster and more scalable inference. However, the approximation must be evaluated carefully, as the accuracy of the results depends on the choice of the family of distributions and the convergence of the algorithm.