Statistics › 🧠 Advanced & Bayesian Modeling

Maximum Likelihood Estimation

The engine under regression and machine learning: pick the parameter that makes your data most plausible.

AdvancedUpper-undergraduate
💡
The big idea: You have data and a model with an unknown knob — a mean, a rate, a probability. Maximum likelihood turns the usual question around: instead of asking how probable the data are for a fixed knob, it asks which knob setting makes the data you actually observed most plausible. Write down the likelihood, take its logarithm to turn a product into a sum, then climb to the peak. For a normal mean that peak sits exactly on the ordinary sample average — which is why the everyday mean is not just a habit but the maximum-likelihood answer.
🎯 By the end, you'll be able to
  • Distinguish a probability (parameter fixed, data varies) from a likelihood (data fixed, parameter varies)
  • Write the likelihood and log-likelihood of an independent sample given a parameter
  • Explain why the maximum likelihood estimate is the peak of the log-likelihood curve
  • Apply the general recipe — write, differentiate, solve — and show the MLE of a normal mean is the sample mean
📎 You should already know
  • Differentiation and finding a maximum by setting the derivative to zero
  • The normal distribution and its density
  • Independent events and multiplying probabilities
  • Logarithms and their properties

Turning the question around

Probability and likelihood use the same formula but ask opposite questions. In a probability question the parameter is fixed and we ask how likely various data are: given a fair coin, how probable is it to see 8 heads in 10 tosses? In a likelihood question the data are fixed — we already saw 8 heads — and we ask which parameter makes what we observed most plausible.

Maximum likelihood estimation, or MLE, is the recipe behind that second question. It is the quiet workhorse under a huge amount of modern statistics — linear and logistic regression, and much of machine learning, all fit their parameters by maximizing a likelihood.

🔑 Likelihood: probability read backwards
The likelihood \( L(\theta) \) is the probability (or density) of the data you actually observed, viewed as a function of the parameter \( \theta \) with the data held fixed. It is not a probability distribution over \( \theta \) — it need not integrate to 1. Same numbers as a probability, opposite variable.

The likelihood of a whole data set

Suppose \( x_1, x_2, \dots, x_n \) are drawn independently from a model with density \( f(x \mid \theta) \). Independence means the probability of seeing the whole batch is the product of the individual densities. Products of many small numbers underflow toward zero and are painful to differentiate, so we take a logarithm: it turns the product into a sum, and because \( \ln \) is strictly increasing it never moves where the maximum is. So we maximize the log-likelihood instead.

\[ L(\theta) = \prod_{i=1}^{n} f(x_i \mid \theta) \]
The likelihood of an independent sample: the product of each observation's density, read as a function of the parameter θ with the data fixed.
\[ \ell(\theta) = \ln L(\theta) = \sum_{i=1}^{n} \ln f(x_i \mid \theta) \]
The log-likelihood turns the product into a sum. Because ln is increasing, its maximizer is identical to the likelihood's — but it is far easier to differentiate.

The MLE is the peak

The maximum likelihood estimate \( \hat\theta \) is the parameter value at the top of the log-likelihood curve — the setting that makes the data you actually saw most plausible:

\( \hat\theta = \arg\max_{\theta}\; \ell(\theta) \)

The tool below fixes a sample of 30 values drawn from a normal model with a known spread \( \sigma \), then plots \( \ell(\mu) \) as you slide the candidate mean \( \mu \). The small ticks along the axis are the data points themselves. Drag \( \mu \): the red dot rides the curve, and the fit is best exactly at the green peak — watch where that peak lands.

🎮 Interactive: climbing the log-likelihood LIVE
The curve is the log-likelihood of a fixed sample as a function of the candidate mean μ. The red marker is your current guess; the green peak is the maximizer. For a normal mean the curve is a downward parabola whose apex sits exactly on the sample mean x-bar. Hit Regenerate to draw a fresh sample and watch the peak move to the new average.
✨ The peak lands on the sample mean
For a normal model the log-likelihood in \( \mu \) is a downward parabola, and its apex sits precisely over the arithmetic mean \( \bar x \) of the data — whatever the known \( \sigma \) happens to be. That is why the everyday average is not merely intuitive: it is the maximum-likelihood estimate of a normal mean. The known \( \sigma \) only controls how sharp the peak is (how much the fit worsens as you move away), not where the peak is.

The general recipe

Almost every MLE problem follows the same three steps:

  1. Write the log-likelihood \( \ell(\theta) \) for your model and data.
  2. Differentiate with respect to the parameter and set the derivative to zero — this is the score equation.
  3. Solve for \( \theta \), and check the solution is a maximum (the second derivative is negative), not a minimum.

Carry that out for the normal mean. The log-likelihood is \( \ell(\mu) = -\tfrac{n}{2}\ln(2\pi\sigma^2) - \tfrac{1}{2\sigma^2}\sum (x_i-\mu)^2 \). Only the last term depends on \( \mu \), so differentiating and setting to zero gives:

\[ \frac{d\ell}{d\mu} = \frac{1}{\sigma^2}\sum_{i=1}^{n}(x_i - \mu) = 0 \;\;\Longrightarrow\;\; \hat\mu = \frac{1}{n}\sum_{i=1}^{n} x_i = \bar x \]
The sum of the deviations is zero exactly when μ equals the sample mean. So the MLE of a normal mean is x-bar — the σ term cancelled out entirely.
📝 Worked example: You model three independent measurements 2, 4, and 9 as coming from a normal distribution with a known standard deviation σ = 3. Find the maximum likelihood estimate of the mean μ.
  1. Write the log-likelihood: \( \ell(\mu) = -\tfrac{n}{2}\ln(2\pi\sigma^2) - \tfrac{1}{2\sigma^2}\sum (x_i-\mu)^2 \). Only the final sum depends on \( \mu \).
  2. Differentiate and set to zero: \( \frac{d\ell}{d\mu} = \frac{1}{\sigma^2}\sum (x_i-\mu) = 0 \), which rearranges to \( \sum x_i - n\mu = 0 \).
  3. Solve: \( \hat\mu = \frac{1}{n}\sum x_i = \frac{2+4+9}{3} = \frac{15}{3} \).
✓ The MLE is μ-hat = 5 — simply the sample mean. The known value σ = 3 never enters the answer; it changes how sharply the log-likelihood peaks, not where the peak sits.

Check your understanding

1. We write L(θ) = P(data | θ) and treat it as a function of θ with the observed data held fixed. This function is called the…
Same formula as a probability, but read as a function of the parameter with the data fixed — that is the likelihood. It need not integrate to 1 over θ.
2. For a normal model with known σ, the maximum likelihood estimate of the mean μ is…
Setting the derivative of the log-likelihood to zero gives Σ(x_i − μ) = 0, so μ-hat = x-bar. The known σ cancels and never affects the location of the peak.
3. Why do we usually maximize the LOG-likelihood rather than the likelihood itself?
ln converts the product of densities into a sum (easier to differentiate) and, because ln is strictly increasing, the value of θ at the peak is unchanged.
4. A sample of five independent values from a normal model with known σ has values summing to 40. The MLE of the mean μ is…
The MLE of a normal mean is the sample mean: x-bar = 40 / 5 = 8. σ affects the sharpness of the peak, not its location, so it is not needed.
✅ Key takeaways
  • A probability fixes the parameter and asks about the data; a likelihood fixes the data and asks about the parameter.
  • The likelihood of an independent sample is the product of its densities; the log-likelihood is the sum of the log-densities and has the same maximizer.
  • The maximum likelihood estimate is the parameter at the peak of the (log-)likelihood curve: θ-hat = argmax ℓ(θ).
  • General recipe: write the log-likelihood, differentiate, set to zero, solve, and confirm it is a maximum.
  • For a normal model with known σ the log-likelihood in μ is a downward parabola whose peak is the sample mean, so the MLE of a normal mean is x-bar.