Statistics › 🧠 Advanced & Bayesian Modeling

Model Selection & Fat Tails

Why a more flexible model always fits the data you have — and often predicts worse on the data you don't.

AdvancedUpper-undergraduate

💡

The big idea: A model that bends to touch every training point has memorized the noise, not the signal. Training error keeps falling as you add complexity, but out-of-sample error follows a U: it drops while the model learns the real pattern, then climbs as it starts chasing random wiggles. Choosing well means paying explicitly for complexity — through validation or penalties like AIC and BIC — and staying alert to fat-tailed data, where the mean and variance you are quietly relying on may not even exist.

🎯 By the end, you'll be able to

Explain why training error is non-increasing in model complexity while out-of-sample error is U-shaped
Decompose expected test error into bias, variance, and irreducible noise, and read the trade-off between the first two
Use a held-out validation set to pick complexity, and read AIC/BIC as fit rewarded minus a complexity penalty
Recognize fat-tailed (e.g. Cauchy) data where the mean or variance may be undefined, and know to plot and robustify instead of trusting averages

📎 You should already know

Least-squares regression and residuals
Expectation, variance, and the normal distribution
The Central Limit Theorem and the law of large numbers
Maximum likelihood (log-likelihood of a fitted model)

Fitting versus memorizing

Give a flexible enough model twenty noisy points and it can thread a curve through every single one, residuals all zero. It looks like a triumph. It is usually a disaster.

The data you fit is one noisy draw from some underlying process: a smooth signal plus random scatter. A curve that hits every point has fit the signal and the scatter. But the scatter is different next time — a fresh sample has its noise in different places — so a model tuned to this sample's noise predicts the next sample worse. This is overfitting: mistaking accidental features of one dataset for structure in the world.

The whole discipline of model selection is about telling those two apart when all you can see is the training data.

🔑 The bias-variance trade-off

Prediction error has two competing sources.

Bias — error from a model too rigid to capture the real pattern (a straight line through a curved trend). High bias means underfitting.
Variance — how much the fitted model jumps around when the training sample changes. A very flexible model swings wildly from one dataset to the next. High variance means overfitting.

Simple models: high bias, low variance. Complex models: low bias, high variance. You cannot drive both to zero at once — the best model lives at the balance point, not at either extreme.

Why training error only ever falls

Here is the trap in one sentence: adding parameters can never increase training error. A degree-4 polynomial contains every degree-3 polynomial as a special case (set the top coefficient to zero), so the best degree-4 fit is at least as good on the training data as the best degree-3 fit. Training error is therefore non-increasing in complexity — it slides toward zero whether or not the extra flexibility is real.

That is exactly why training error cannot be your judge. It rewards complexity unconditionally. To see whether flexibility is genuine, you must test the model on data it did not fit.

\[ \underbrace{\mathbb{E}\big[(y-\hat f(x))^2\big]}_{\text{expected test error}} \;=\; \underbrace{\big(\mathbb{E}[\hat f(x)]-f(x)\big)^2}_{\text{bias}^2} \;+\; \underbrace{\operatorname{Var}[\hat f(x)]}_{\text{variance}} \;+\; \sigma^2 \]

Expected squared prediction error at a point splits into bias squared, variance, and the irreducible noise σ² baked into the data. You can trade bias against variance; σ² is a floor no model can beat.

Watch the U-curve form

The sim below generates about twenty noisy points from a fixed smooth truth, plus a separate held-out validation set drawn from the same truth (shown as hollow circles). Slide the polynomial degree from 0 up to 7 and fit least squares at each degree.

Watch two things at once. In the top panel the fitted curve goes from too stiff (a flat or straight line missing the trend) to sensible to visibly frantic — wiggling to touch training points and overshooting between them. In the bottom panel the two error curves tell the story numerically: the training MSE falls monotonically, while the validation MSE traces a U — down, a minimum at some moderate degree, then up as overfitting sets in. Hit Regenerate data a few times: the exact minimum wanders, which is itself the point — the best degree is a property of signal-plus-noise, not a fixed constant.

🎮 Interactive: overfitting and the training-vs-validation U-curve LIVE

Top: the noisy data (filled points = training, hollow = held-out validation) with the degree-you-choose least-squares fit. Bottom: training MSE (falls forever) and validation MSE (a U). The validation minimum marks the sweet spot; degrees to its right are overfitting.

✨ The training curve lies; the validation curve tells the truth

Notice that at high degree the training MSE is near its smallest while the validation MSE is climbing back up. If you had only the training number you would happily pick the most complex model — and predict worst. The gap between the falling training curve and the rising validation curve largely reflects the variance term of the decomposition, made visible: the more the flexible model would swing from one training sample to the next, the more its rosy training score overstates how it does on fresh data. The honest estimate of how a model will do on new data comes from data it never touched.

Paying for complexity without a validation set

Holding out data works, but it costs data and it is noisy for small samples. An alternative is to score a model by its fit minus an explicit penalty for its size. Two classic scores do exactly this, using the maximized log-likelihood \(\ln\hat L\), the number of free parameters \(k\), and the sample size \(n\):

\[ \mathrm{AIC} = 2k - 2\ln\hat L, \qquad\qquad \mathrm{BIC} = k\ln n - 2\ln\hat L \]

Both reward fit (the −2 ln L̂ term shrinks as fit improves) and charge for parameters. Lower is better. BIC's per-parameter charge is ln n, which exceeds AIC's flat 2 whenever n > 7 — so BIC leans harder toward simpler models as data grows.

Fat tails: when the mean itself goes missing

All of this quietly assumes the noise has a finite mean and variance — that averages settle down. For fat-tailed data, where extreme values are far more common than a bell curve allows, that assumption can fail outright. The textbook warning is the Cauchy distribution, whose tails fall off like \(1/x^2\):

\[ f(x) = \frac{1}{\pi}\cdot\frac{1}{1+x^2} \]

The standard Cauchy density. It looks bell-ish near the center, but its tails are so heavy that the integral defining the mean does not converge — the mean is undefined and the variance is infinite.

⚠️ Averaging can stop working

Sample from a Cauchy and keep averaging more draws and the sample mean does not converge — the average of Cauchy draws is itself Cauchy, just as wild with a million points as with one. The Central Limit Theorem and the law of large numbers simply do not apply, because they require a finite variance the Cauchy lacks. Real fat-tailed data (some financial returns, network traffic, city sizes) can behave the same way: a single record-breaking observation moves the mean by more than thousands of ordinary ones. Defenses: plot the data before trusting any summary, prefer robust statistics (median and interquartile range over mean and standard deviation), and be wary of variance-based tools — including AIC's Gaussian-likelihood assumptions — on data whose tails you have not checked.

📝 Worked example: You fit two models to the same n = 100 observations. Model A has k = 3 parameters and maximized log-likelihood ln L̂ = −140. Model B is richer: k = 6 parameters and ln L̂ = −135. Which does AIC prefer, and which does BIC prefer?

AIC for A: \(2k - 2\ln\hat L = 2(3) - 2(-140) = 6 + 280 = 286\).
AIC for B: \(2(6) - 2(-135) = 12 + 270 = 282\). Lower is better, so AIC picks the richer Model B (282 < 286): its 5-point likelihood gain outweighs a penalty of only 2 per extra parameter.
BIC uses \(\ln n = \ln 100 \approx 4.605\) per parameter. BIC for A: \(3(4.605) + 280 = 13.82 + 280 = 293.82\).
BIC for B: \(6(4.605) + 270 = 27.63 + 270 = 297.63\). Now the heavier per-parameter charge tips the balance the other way.

✓ AIC prefers the more complex Model B (282 vs 286); BIC prefers the simpler Model A (293.82 vs 297.63). The disagreement is the lesson: because BIC charges ln n per parameter versus AIC's flat 2, BIC is stingier with complexity — and the two criteria can, and often do, point at different models.

Check your understanding

1. As you increase model complexity (say, polynomial degree) fitted by least squares, the TRAINING error…

A more complex model nests the simpler ones as special cases, so its best training fit is at least as good. Training error slides toward zero regardless of whether the added flexibility is real — which is why it cannot judge a model.

2. The out-of-sample (validation) error as complexity increases is best described as…

Falling bias dominates at first, then rising variance takes over. The minimum of that U is the sweet spot; degrees to its right chase noise and predict worse.

3. On the same data with n = 100, which criterion penalizes each extra parameter MORE heavily?

AIC charges a flat 2 per parameter; BIC charges ln n. With n = 100, ln 100 ≈ 4.6 > 2, so BIC penalizes complexity harder and leans toward simpler models.

4. You keep drawing samples from a Cauchy (fat-tailed) distribution and averaging more and more of them. The sample mean…

The CLT and law of large numbers need a finite variance, which the Cauchy lacks. The mean of Cauchy draws is again Cauchy, so more data does not tame it — a reason to plot the data and use robust statistics like the median.

✅ Key takeaways

Training error is non-increasing in complexity, so it always rewards more parameters and can never, on its own, tell you when a model is overfitting.
Out-of-sample error is U-shaped: bias falls then variance rises, and expected test error = bias² + variance + irreducible noise σ².
A held-out validation set (as in the sim) estimates true prediction error; its minimum, not the training minimum, marks the right complexity.
AIC = 2k − 2 ln L̂ and BIC = k ln n − 2 ln L̂ reward fit and charge for parameters; BIC's ln n penalty is stricter than AIC's flat 2, so they can disagree.
Fat-tailed data (e.g. Cauchy, tails ~1/x²) can have an undefined mean and infinite variance, breaking the CLT and averages — plot the data and prefer robust statistics like the median.