Maximum Likelihood Estimation (MLE) is a statistical technique used to estimate the values of parameters that maximize the likelihood of observing the given dataset. Essentially, MLE helps us find the parameters that best explain the data by maximizing a likelihood function.
Here are the main principles of MLE:
1. **Probability Distribution:** MLE assumes that there is an underlying probability distribution that explains the data. Commonly assumed probability distributions include Normal, Exponential, and Poisson, among others.
2. **Likelihood Function:** The likelihood function is the probability of observing the data given the parameters of the distribution. It is usually denoted as L(θ|x) or P(x|θ), where θ is the vector of parameters to be estimated, and x is the observed data. The goal of MLE is to find the parameter values that maximize the likelihood function.
3. **Log-likelihood:** Since the likelihood function can often be a complicated and small number due to the product of probabilities, taking the natural logarithm of the likelihood function (log-likelihood) is often more convenient. Mathematically, it is defined as l(θ|x) = ln L(θ|x). Note that maximizing the log-likelihood function is equivalent to maximizing the likelihood function.
4. **Optimization:** MLE involves solving an optimization problem, usually maximizing the log-likelihood function with respect to the parameters. This can be achieved using various optimization techniques, such as gradient ascent, Newton-Raphson method, and Expectation-Maximization (EM) algorithm.
Let’s illustrate MLE with an example. Suppose we have observed the following dataset of five independent and identically distributed data points, x1, x2, …, x5, and we assume that they are drawn from a normal distribution. Our task is to estimate the mean μ and the variance σ2 of that normal distribution.
Given a normal distribution N(μ, σ2), the probability density function (pdf) is:
$$f(x_i|\mu, \sigma^2) = \frac{1}{\sqrt{2\pi \sigma^2}}\exp\left\{-\frac{(x_i - \mu)^2}{2\sigma^2} \right\}$$
Since the data points are assumed to be independent and identically distributed (i.i.d.), the likelihood function is the product of the pdfs:
$$L(\mu, \sigma^2|x) = \prod_{i=1}^5 f(x_i|\mu, \sigma^2)$$
To make calculations more convenient, we compute the log-likelihood function:
$$l(\mu, \sigma^2|x) = \sum_{i=1}^5 \ln f(x_i|\mu, \sigma^2) = -\frac{5}{2}\ln\left(2\pi\sigma^2\right) - \sum_{i=1}^5\frac{(x_i - \mu)^2}{2\sigma^2}$$
Now, we need to find the values of μ and σ2 that maximize this log-likelihood function. We can do this by taking the partial derivatives of l(μ, σ2|x) with respect to μ and σ2 and setting them to zero:
$$\frac{\partial l}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^5 (x_i - \mu) \quad \text{and} \quad \frac{\partial l}{\partial \sigma^2} = -\frac{5}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^5 (x_i - \mu)^2$$
After solving these equations, we would obtain the MLE of μ and σ2. These estimates can then be used to make further inferences or predictions based on the normal distribution.