What is the Law of Large Numbers?

The Law of Large Numbers (LLN) is a fundamental theorem in probability and statistics that describes the result of performing the same experiment a large number of times. More specifically, it states that as the number of identically distributed, independently obtained random variables (sample size) increases, the sample mean converges to the expected value, or true mean, of the underlying distribution. This theorem is important in understanding the long-term stability of statistical averages of random variables.

There are two main versions of the Law of Large Numbers: the Weak Law and the Strong Law.

1. Weak Law of Large Numbers (WLLN): Given a sequence of independent and identically distributed (i.i.d.) random variables, X₁, X₂, …, X_n, each with finite mean μ = E[X_i], the sample mean $\overline{X_n}$ converges in probability to the true mean as n → ∞. Mathematically, WLLN states that, for any ϵ > 0:

$$\lim_{n\to\infty} P\left( \left|\overline{X_n} - \mu\right| \ge \epsilon\right) = 0$$

2. Strong Law of Large Numbers (SLLN): The strong version extends the weak version by stating that the sample mean almost surely converges to the true mean. More formally, for the i.i.d. random variables X₁, X₂, …, X_n and their sample mean $\overline{X_n}$,

$$P\left( \lim_{n\to\infty} \overline{X_n} = \mu\right) = 1$$

To illustrate the Law of Large Numbers, let’s use the example of flipping a fair coin. Consider the random variable X_i as the outcome of the i-th coin flip, with X_i = 1 for a head and X_i = 0 for a tail. The true mean (expected value) for a fair coin is μ = E[X_i] = 0.5, since there’s an equal chance of obtaining a head and a tail.

Now, imagine that we flip this coin a large number of times, say n = 10⁵. According to the LLN, as n becomes very large, the sample mean, which is the ratio of the number of heads to the total number of flips, will approach the true mean of the underlying distribution (which is 0.5 for a fair coin).

As the number of flips keeps increasing, the deviation from the expected value of 0.5 will decrease, meaning the proportion of the observed outcomes of heads will converge to the true mean, 0.5.

In conclusion, the Law of Large Numbers is a fundamental concept in probability and statistics that establishes the relation between the sample mean and the true mean as the number of trials increases. It is an essential theorem for understanding long-term trends in random processes and justifies the use of sample estimates for making inferences about population parameters.