In Bayesian statistics, a prior distribution and a posterior distribution represent different stages of the process of updating our beliefs based on observed data.
A prior distribution, denoted by P(θ), characterizes our prior knowledge or beliefs about the unknown parameter θ before observing any data. This distribution is usually based on expert opinion, existing studies, or even just an educated guess. In many cases, we choose a conjugate prior for mathematical convenience.
A posterior distribution, denoted by P(θ|X), is the updated distribution of the parameter θ after taking into account the observed data X. It represents our revised beliefs after incorporating the evidence provided by the data.
To move from prior to posterior distribution, we use Bayes’ theorem, which states:
$$P(\theta | X) = \frac{P(X | \theta)P(\theta)}{P(X)}$$
Where:
- P(θ|X): Posterior distribution, the probability of θ given the observed data X
- P(X|θ): Likelihood function, the likelihood of observing the data X given the parameter θ
- P(θ): Prior distribution, the initial probability distribution of θ, before considering the data
- P(X): Marginal likelihood, the probability of observing data X across all possible values of θ, which acts as a normalizing constant
Let’s illustrate this with an example:
Suppose we want to estimate the probability of success (p) of a new marketing strategy, based on the number of successful campaigns (x) out of total campaigns (n). We start with a prior belief on the probability of success, say a Beta distribution:
$$P(p) = Beta(p; \alpha, \beta) = \frac{1}{B(\alpha, \beta)} p^{\alpha - 1} (1 - p)^{\beta - 1}$$
Here, α and β are the shape parameters, and B is the Beta function that normalizes the distribution.
Now assume we run the new strategy on n campaigns, and observe x successful ones. The binomial likelihood function is:
$$P(X | p) = \binom{n}{x} p^x (1 - p)^{n - x}$$
According to Bayes’ theorem, the posterior distribution is given by:
$$P(p | X) \propto P(X | p) P(p) = \binom{n}{x} p^x (1 - p)^{n - x} \cdot \frac{1}{B(\alpha, \beta)} p^{\alpha - 1} (1 - p)^{\beta - 1}$$
Simplifying, we get the posterior distribution as:
P(p|X) ∝ px + α − 1(1 − p)n − x + β − 1
Notice that the posterior distribution is also a Beta distribution, with updated parameters α′ = x + α and β′ = n − x + β. The Beta prior is a conjugate prior of the binomial likelihood, which makes the update rule and the computation of the posterior distribution easier.
In summary, the main difference between a prior and a posterior distribution is that a prior distribution represents our initial beliefs about an unknown parameter, whereas the posterior distribution represents our updated beliefs after incorporating the observed data. The relationship between these two distributions is defined by Bayes’ theorem and the likelihood function.