A model trained to predict children's reading scores from their shoe size fits the training data almost perfectly — a suspiciously elegant result, until you remember that shoe size does not cause reading ability. Both rise with age. The model has latched onto a spurious correlation: a relationship that holds in the data you collected but reflects no real connection between cause and effect.
Spurious correlations are the seductive failure of machine learning. The model is doing exactly what you asked — finding the feature that tracks the label — and it finds one that tracks it beautifully. The trap closes later, when you deploy on people whose shoe size has stopped growing but whose reading keeps improving, and the confident model turns out to have been measuring the wrong thing all along.
flowchart TD C["hidden cause: age"] --> A["shoe size grows"] C --> B["reading score grows"] A -.->|"correlated,<br/>not causal"| B style C fill:#3776ab,color:#fff
The confounder behind the curtain
The reason shoe size and reading score move together is a third variable — age — that drives both of them. Age is a confounder: a hidden cause that creates an apparent link between two things, neither of which causes the other. Because the confounder was never recorded as a feature, the model has no way to attribute the pattern to it, so it attributes the pattern to the only thing it can see: shoe size.
Correlation is a measurement of how two variables move together, nothing more. It cannot tell you that one causes the other, and it cannot reveal that a hidden third variable is pulling both strings. A high correlation is a reason to look closer, never a reason to conclude.
import math
def pearson(xs, ys):
n = len(xs)
mx = sum(xs) / n
my = sum(ys) / n
num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
dx = math.sqrt(sum((x - mx) ** 2 for x in xs))
dy = math.sqrt(sum((y - my) ** 2 for y in ys))
return num / (dx * dy)
shoe = [5, 6, 7, 8, 9]
score = [10, 15, 25, 30, 40]
print('correlation:', round(pearson(shoe, score), 2))
Run it and the correlation prints 0.99 — as close to a perfect link as real data ever shows. By every statistical measure, shoe size is a wonderful predictor of reading score. The number is correct; the conclusion is not. The correlation is real, but it is carried entirely by the hidden cause, age, which grew both variables together. Remove age's influence and the link between shoe size and reading vanishes. The model cannot know that, because age never made it into the feature set, so it trusts the 0.99 and builds on sand.
flowchart LR T["training: shoe tracks score"] --> M["model: score from shoe"] M --> D["deploy on adults:<br/>shoe flat, score rises"] D --> F["predictions collapse"] style F fill:#b91c1c,color:#fff
import math
def fit_line(xs, ys):
n = len(xs)
mx = sum(xs) / n
my = sum(ys) / n
a = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
b = my - a * mx
return a, b
def rmse(xs, ys, a, b):
err = [(a * x + b - y) ** 2 for x, y in zip(xs, ys)]
return math.sqrt(sum(err) / len(xs))
shoe_train, score_train = [5, 6, 7, 8, 9], [10, 15, 25, 30, 40]
a, b = fit_line(shoe_train, score_train)
shoe_adult, score_adult = [9, 9, 10], [60, 70, 80]
print('train RMSE:', round(rmse(shoe_train, score_train, a, b), 2))
print('adult RMSE:', round(rmse(shoe_adult, score_adult, a, b), 2))
Why the model looked brilliant and then failed
On the training data the line fits almost perfectly — an error of about one point. By every in-sample measure the model is a success, and anyone reviewing only the training score would approve it. The disaster appears only on the adult population, where the error balloons roughly thirty-fold. Nothing changed in the mathematics; what changed is the relationship between the feature and the label. In children, shoe size and reading score climb together; in adults, shoe size is flat while reading keeps rising, and the line that connected the two in children has nothing left to hold onto.
This is the signature of a spurious feature: it predicts well wherever the coincidence holds, and it falls apart wherever the coincidence breaks. The collapse is the proof that the original correlation was never causal.
flowchart LR
H["a strong correlation"] --> Q{"causal, or<br/>a hidden cause?"}
Q -->|"causal"| U["use the feature"]
Q -->|"hidden cause"| G["test on data from<br/>a different regime"]
G --> V["does it still predict?"]
style U fill:#166534,color:#fff
style V fill:#b45309,color:#fff
Ice cream sales and drownings rise and fall together over the year. The best explanation is:
Hot summer weather raises both ice cream consumption and the number of people swimming, so drownings rise too. Neither variable causes the other; the shared cause is the season.
What does this print? Pearson correlation between shoe size and reading score.
import math xs = [5, 6, 7, 8, 9] ys = [10, 15, 25, 30, 40] mx = sum(xs) / len(xs) my = sum(ys) / len(ys) num = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) den = math.sqrt(sum((x - mx) ** 2 for x in xs)) * math.sqrt(sum((y - my) ** 2 for y in ys)) print(round(num / den, 2))
Both lists rise together, so the correlation is positive and near 1.
Rounding to two decimals gives 0.99.
Write pearson(xs, ys) returning the Pearson correlation coefficient for two equal-length lists, computed as sum((x-mx)(y-my)) / (sqrt(sum((x-mx)^2)) * sqrt(sum((y-my)^2))).
import math
def pearson(xs, ys):
# covariance over the product of the two standard deviations (population form)
pass
Perfect agreement gives 1.0; perfect opposition gives -1.0.
Use the population form (divide the squared deviations by n implicitly via sum).
import math
def pearson(xs, ys):
n = len(xs)
mx = sum(xs) / n
my = sum(ys) / n
num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
den = math.sqrt(sum((x - mx) ** 2 for x in xs)) * math.sqrt(sum((y - my) ** 2 for y in ys))
return num / den
Write fit_line(xs, ys) returning the slope a and intercept b of the least-squares line y = a*x + b for two equal-length lists, as a tuple (a, b).
def fit_line(xs, ys):
# least-squares slope a and intercept b; return (a, b)
pass
Slope a is the sum of cross-deviations divided by the sum of squared x-deviations.
Intercept b is my - a * mx.
def fit_line(xs, ys):
n = len(xs)
mx = sum(xs) / n
my = sum(ys) / n
a = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
b = my - a * mx
return a, b
A feature has a 0.98 correlation with the label on your training set. Before trusting it, what should you do?
A high correlation earns scrutiny, not trust. Ask whether the feature could cause the label, consider what hidden variable might drive both, and confirm the relationship still holds on data drawn from a different regime where any coincidence would break.
Recap
- A spurious correlation is a strong link between a feature and the label that holds in your data but has no causal basis, so it can vanish without warning.
- A confounder is a hidden cause that drives two variables to move together — age drives both shoe size and reading score — and the model blames the visible one.
- Correlation measures movement, not causation; a high value is a prompt to investigate, never a conclusion.
- A spurious model fits training data beautifully and collapses on a different regime, where the coincidence no longer holds — the collapse is the proof it was never causal.
- The strongest defence is domain knowledge: a human who understands the mechanism can reject a link the correlation alone would celebrate.
That closes the module on how models deceive you — by cheating, by going stale, and by mistaking coincidence for cause. Next you will turn to keeping them honest when the classes are wildly uneven.