Statistics › 🧠 Advanced & Bayesian Modeling

Moment Generating Functions

One function that quietly stores every moment of a distribution — and hands you the mean back as its slope at zero.

AdvancedUpper-undergraduate probability
💡
The big idea: The moment generating function M(t) = E[e^{tX}] is a compact fingerprint of a random variable. Differentiate it at t = 0 and out fall the moments in order: the first derivative is the mean, the second gives you the variance, and so on. Two variables with the same MGF must have the same distribution, and the MGF of a sum of independent variables is just the product of their MGFs — which is why it is one of the sharpest tools in probability.
🎯 By the end, you'll be able to
  • Define the moment generating function M(t) = E[e^{tX}] and explain why it is called moment generating
  • Recover the mean and variance from M'(0) and M''(0)
  • State and use the MGFs of the Bernoulli, normal, and exponential distributions
  • Apply the uniqueness property and the sum rule M_{X+Y} = M_X M_Y for independent variables
📎 You should already know
  • Expected value and variance of a random variable
  • Taylor series and differentiation
  • The normal, Bernoulli, and exponential distributions

A single function that remembers everything

Suppose you wanted to store all the information about a random variable \(X\) — its mean, its variance, its skewness, every moment \(E[X^k]\) — in one object you could carry around. The moment generating function (MGF) does exactly that. Instead of tracking an infinite list of moments, you track one function of a helper variable \(t\).

The trick is to average the exponential \(e^{tX}\). Because \(e^{tX}\) expands into a power series in \(X\), averaging it folds every power of \(X\) — every moment — into a single tidy expression. Wiggle the knob \(t\), differentiate, and the moments drop out one at a time.

\[ M_X(t) = E\!\left[e^{tX}\right] = \begin{cases} \displaystyle\sum_x e^{tx}\,p(x) & X \text{ discrete} \\[1.2em] \displaystyle\int_{-\infty}^{\infty} e^{tx}\,f(x)\,dx & X \text{ continuous} \end{cases} \]
The MGF is just an expected value — a weighted average of e^{tx} over the distribution of X. It is a function of the dummy variable t, not of X.
🔑 Why it 'generates' moments

Expand the exponential inside the expectation using its Taylor series \(e^{tX} = 1 + tX + \tfrac{(tX)^2}{2!} + \cdots\), then take the expectation term by term:

\(M(t) = 1 + t\,E[X] + \tfrac{t^2}{2!}E[X^2] + \tfrac{t^3}{3!}E[X^3] + \cdots\)

So the moments \(E[X^k]\) are literally the coefficients of this series (each scaled by \(k!\)). Reading a coefficient off a power series is what differentiating at zero does — which is why the derivatives of \(M\) at \(t=0\) hand you the moments.

\[ M^{(k)}(0) = E\!\left[X^k\right], \qquad M'(0) = E[X], \qquad M''(0) = E\!\left[X^2\right] \]
Differentiate k times and set t = 0 to read off the k-th moment. Note M(0) = E[e^0] = 1 for every distribution.
\[ \operatorname{Var}(X) = E\!\left[X^2\right] - \big(E[X]\big)^2 = M''(0) - \big(M'(0)\big)^2 \]
Variance from the first two derivatives at zero: the second derivative minus the square of the first.

Watch the slope become the mean

The tool below plots \(M(t)\) for three distributions with clean closed-form MGFs. Every curve passes through the point \((0, 1)\) because \(M(0) = 1\). The dashed line is the tangent at that point, and its slope is the numeric \(M'(0)\).

Pick a distribution and change its parameter. The readout compares the measured slope \(M'(0)\) against the known mean \(E[X]\) — they track each other exactly. For the exponential, notice the curve blows up as \(t\) approaches \(\lambda\): the MGF only exists for \(t < \lambda\), and the tool stops the plot before that pole.

🎮 Interactive: M(t) and the tangent at zero LIVE
The MGF curve M(t) for Bernoulli, standard normal, and exponential distributions. The dashed tangent at t = 0 has slope M'(0); the readout shows it equals the known mean E[X]. For the exponential the plot halts below the pole at t = lambda, where the MGF ceases to exist.
✨ The mean is a slope; the variance is a curvature

Because \(M(0) = 1\) is fixed for everyone, the distributions differ in how they leave that point. The initial slope is the mean \(E[X]\), and the curvature at zero encodes the second moment \(E[X^2]\). A steeper, more sharply bending curve near \(t=0\) is a variable with a larger mean and spread. So the whole shape of \(X\) is written into how \(M\) departs from \((0,1)\).

Common MGFs, uniqueness, and sums

A few distributions have MGFs worth memorizing:

  • Bernoulli(p): \(M(t) = 1 - p + p\,e^{t}\), giving \(E[X] = p\).
  • Standard normal: \(M(t) = e^{t^2/2}\), giving \(E[X] = 0\), and more generally \(N(\mu, \sigma^2)\) has \(M(t) = e^{\mu t + \sigma^2 t^2 / 2}\).
  • Exponential(λ): \(M(t) = \dfrac{\lambda}{\lambda - t}\) for \(t < \lambda\), giving \(E[X] = 1/\lambda\).

Two properties make MGFs powerful. Uniqueness: if two variables have the same MGF on an interval around \(t = 0\), they have the same distribution — the MGF pins a distribution down completely. The sum rule: for independent \(X\) and \(Y\), the MGF of the sum is the product of the individual MGFs. Together these turn hard convolution problems into easy multiplication: multiply the MGFs, then recognize the result. (One caveat for honesty: not every distribution has an MGF — heavy-tailed variables like the Cauchy have no interval where the integral converges.)

\[ M_{X+Y}(t) = M_X(t)\,M_Y(t) \qquad (X, Y \text{ independent}) \]
The MGF of a sum of independent variables is the product of their MGFs — because e^{t(X+Y)} = e^{tX}e^{tY} and independence lets the expectation split.
📝 Worked example: For an Exponential(λ) variable the MGF is M(t) = λ / (λ − t), valid for t < λ. Use it to find E[X] and Var(X).
  1. First derivative: \(M'(t) = \dfrac{d}{dt}\,\lambda(\lambda - t)^{-1} = \dfrac{\lambda}{(\lambda - t)^2}\). At \(t = 0\): \(M'(0) = \lambda / \lambda^2 = 1/\lambda\). So \(E[X] = 1/\lambda\).
  2. Second derivative: \(M''(t) = \dfrac{2\lambda}{(\lambda - t)^3}\). At \(t = 0\): \(M''(0) = 2\lambda / \lambda^3 = 2/\lambda^2\). So \(E[X^2] = 2/\lambda^2\).
  3. Variance: \(\operatorname{Var}(X) = M''(0) - (M'(0))^2 = \dfrac{2}{\lambda^2} - \dfrac{1}{\lambda^2} = \dfrac{1}{\lambda^2}\).
✓ E[X] = 1/λ and Var(X) = 1/λ², matching the standard exponential formulas — recovered from the MGF alone, with no integration of x·f(x).

Check your understanding

1. For any random variable with an MGF, the derivative M'(0) equals…
Differentiating the series M(t) = 1 + tE[X] + (t^2/2)E[X^2] + … once and setting t = 0 leaves E[X]. (M(0) itself is always 1.)
2. Given M'(0) = E[X] and M''(0) = E[X^2], the variance of X is…
Var(X) = E[X^2] − (E[X])^2 = M''(0) − (M'(0))^2.
3. If X and Y are independent, the MGF of their sum X + Y is…
Since e^{t(X+Y)} = e^{tX}e^{tY} and independence lets the expectation factor, M_{X+Y}(t) = M_X(t)·M_Y(t) — a product, not a sum.
4. For Exponential(λ) with M(t) = λ/(λ − t), the mean E[X] = M'(0) is…
M'(t) = λ/(λ − t)^2, so M'(0) = λ/λ^2 = 1/λ.
✅ Key takeaways
  • The moment generating function is M(t) = E[e^{tX}] — an average of e^{tx} that stores every moment of X in one function.
  • Its derivatives at zero recover the moments: M'(0) = E[X] and M''(0) = E[X^2], while M(0) = 1 always.
  • Variance comes straight from the MGF: Var(X) = M''(0) − (M'(0))^2.
  • Memorable MGFs: Bernoulli(p) = 1 − p + pe^t, standard normal = e^{t^2/2}, Exponential(λ) = λ/(λ − t) for t < λ.
  • Uniqueness: equal MGFs mean equal distributions. Sum rule: for independent X and Y, M_{X+Y}(t) = M_X(t)·M_Y(t).