What are moment generating functions?

Moment generating functions (MGFs) are important mathematical tools often used in statistics and probability theory to derive the moments of a random variable. In essence, an MGF is a function that encodes information about all moments of a given probability distribution, making it a convenient way to work with characteristic functions and moment-based properties.

Formally, the moment generating function M_X(t) of a random variable X is defined as the expected value of e^tX, where t is a scalar parameter. For continuous random variables with a probability density function (PDF) f(x), the MGF can be written as an integral:

M_X(t) = 𝔼[e^tX] = ∫_− ∞^∞e^txf(x)dx

For discrete random variables with probability mass function (PMF) p(x), the MGF is given by a sum:

M_X(t) = 𝔼[e^tX] = ∑_xe^txp(x)

The key property of MGFs is that their derivatives evaluated at t = 0 provide the moments of the distribution. For instance, the n-th derivative of the MGF evaluated at t = 0 gives the n-th moment about the origin:

$$\frac{d^n}{dt^n} M_X(t) \Big|_{t=0} = \mathbb{E}\left[X^n\right]$$

As an example, consider the standard normal distribution, which has a PDF given by:

$$f(x) = \frac{1}{\sqrt{2 \pi}} e^{-\frac{x^2}{2}}$$

The MGF of the standard normal distribution can be computed as:

$$M_X(t) = \mathbb{E}\left[e^{tX}\right] = \int_{-\infty}^{\infty} e^{tx}\frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}} dx = e^{\frac{t^2}{2}}$$

Now, let’s find the first and second moments of this distribution using the MGF:

$$\frac{d}{dt} M_X(t) \Big|_{t=0} = \left(\frac{2t}{2} e^{\frac{t^2}{2}}\right) \Big|_{t=0} = 0$$

$$\frac{d^2}{dt^2} M_X(t) \Big|_{t=0} = \left( e^{\frac{t^2}{2}} + t^2 e^{\frac{t^2}{2}} \right) \Big|_{t=0} = 1$$

This confirms our knowledge of the moments of the standard normal distribution: the first moment (mean) is 0, and the second moment (variance) is 1.

In summary, moment generating functions are powerful mathematical tools that provide a compact representation of the moments of a random variable. They simplify the analysis of probability distributions by directly encoding the moments and allowing for easy derivations of moment-based properties.