A new kidney-stone treatment posts a higher recovery rate than the old one, so the hospital prepares to switch every patient over. Then a clinician splits the results by stone size, and the picture flips: inside both the small-stone and large-stone groups, the old treatment wins. Aggregated one way the new treatment looks superior; split by a single variable, it loses everywhere.
That reversal is Simpson's paradox, and it is not a trick of statistics — it is what the data actually does when a hidden variable is quietly carrying the comparison. The same pattern surfaces in hiring funnels and A/B tests wherever a rate averages over unbalanced groups.
The stakes are decision-shaped. A hospital that trusts only the headline number adopts the worse treatment. The cure is not a more sophisticated model; it is the discipline of stratifying — looking inside the groups — before you decide.
flowchart TD H["aggregate rate: A 0.758, B 0.803"] --> Q["who wins overall?"] Q -->|"B looks better"| DecideB["adopt B, the wrong call"] Q -.-> G["split by stone size"] G --> S["small: A 0.90, B 0.87"] G --> L["large: A 0.71, B 0.61"] S --> DecideA["A wins both groups"] L --> DecideA style DecideB fill:#b91c1c,color:#fff style DecideA fill:#166534,color:#fff
The numbers, plainly
Recovery counts for two treatments across two stone sizes. Each rate is simply recovered divided by treated.
- Treatment A: 81 of 90 small-stone patients recovered (90%); 192 of 270 large-stone patients (71%).
- Treatment B: 234 of 270 small-stone patients (87%); 55 of 90 large-stone patients (61%).
Pool each treatment's counts together and the overall rate reverses: A lands at 76%, B at 80%. B wins the headline and loses every subgroup it is actually measured in.
results = {
'A': {'small': (81, 90), 'large': (192, 270)},
'B': {'small': (234, 270), 'large': (55, 90)},
}
for t in ('A', 'B'):
r = results[t]
rec = r['small'][0] + r['large'][0]
tot = r['small'][1] + r['large'][1]
print('Treatment', t, 'overall:', round(rec / tot, 3))
The headline favours B
Run those pooled numbers and Treatment B prints the higher overall recovery rate, 0.803 to A's 0.758. On that evidence alone B is the better treatment, and most dashboards would stop right there. The aggregate rate is easy to compute, easy to put on a slide, and — when a confounder is hiding inside it — quietly wrong. Before you trust an overall rate, you have to ask what mixture it is averaging over.
results = {
'A': {'small': (81, 90), 'large': (192, 270)},
'B': {'small': (234, 270), 'large': (55, 90)},
}
for t in ('A', 'B'):
for g in ('small', 'large'):
rec, tot = results[t][g]
print('Treatment', t, g, 'stone:', round(rec / tot, 3))
flowchart LR Mix["case mix handed to each"] --> ABox["A got 270 large, 90 small"] Mix --> BBox["B got 90 large, 270 small"] ABox --> DragA["A dragged down by hard cases"] BBox --> LiftB["B lifted by easy cases"] style DragA fill:#b45309,color:#fff style LiftB fill:#166534,color:#fff
Why the better treatment can lose overall
The paradox needs one ingredient: the groups are distributed unevenly across the two treatments. Treatment A was given far more large-stone patients — 270 of its 360 cases — and large stones recover less often no matter which treatment you use. Treatment B was given mostly small stones — 270 of its 360 — which recover almost regardless of treatment.
A's overall rate is therefore dragged down by the hard cases it was handed, while B's is lifted by the easy ones. A pooled rate is a weighted average, and here the weights — the share of small versus large stones — differed between treatments. Average two treatments over different mixtures and the average can point the wrong way.
Hold the mixture fixed and the lie disappears
Give both treatments the same mixture of stones — 180 small and 180 large each — and apply each treatment's own per-group rates. A now wins overall, because its rates are higher in both groups and nothing is tilting the weights. Same recovery rates, same mixture, correct conclusion.
The paradox never came from the rates themselves; it came from comparing rates that had been averaged over different mixtures. Forcing a common mixture is exactly what stratifying, or re-weighting to a shared standard, does — and the runnable block below does it in a few lines.
a_rates = {'small': 0.90, 'large': 0.711}
b_rates = {'small': 0.867, 'large': 0.611}
common_mix = {'small': 180, 'large': 180}
def weighted(rates, mix):
return sum(rates[g] * mix[g] for g in mix) / sum(mix.values())
print('A re-weighted:', round(weighted(a_rates, common_mix), 3))
print('B re-weighted:', round(weighted(b_rates, common_mix), 3))
Spotting it before it bites
Simpson's paradox cannot hide if you ask one question of every comparison: do the groups have the same composition? When the answer is no — one treatment saw more severe cases, one variant reached more new users, one store sat in a pricier region — a headline rate is no longer safe to compare directly. Compute the rate within each group first; only when the per-group answers agree does the aggregate tell the truth.
The tell is always the same shape: a comparison that reverses, or weakens sharply, the moment you condition on a third variable. When you see that, you have found a confounder, and the aggregate is the one number you should not report on its own.
flowchart TD Start["new patient arrives"] --> Ask["what is the stone size?"] Ask -->|"small"| Small["use the small-group rates"] Ask -->|"large"| Large["use the large-group rates"] Small --> PickA["A has the higher rate: pick A"] Large --> PickA2["A has the higher rate: pick A"] style PickA fill:#166534,color:#fff style PickA2 fill:#166534,color:#fff
What does this print? The overall rate pools both stone-size groups for Treatment A.
recovered = 81 + 192 total = 90 + 270 print(round(recovered / total, 3))
Recovered = 81 + 192 = 273. Treated = 90 + 270 = 360.
273 / 360 = 0.7583..., which rounds to 0.758.
Write success_rate(recovered, total) that returns the recovered fraction, recovered / total.
def success_rate(recovered, total):
# recovered divided by total
pass
One expression: recovered divided by total.
Integer division in Python 3 with / already returns a float.
def success_rate(recovered, total):
return recovered / total
This overall_rate is meant to pool the raw counts across both groups, but it averages the two group rates instead — a classic mistake that weights the small and large groups equally even when their sizes differ. Fix it to pool the recovered and total counts, then divide.
def overall_rate(results):
# results is {'small': (rec, tot), 'large': (rec, tot)}
small = results['small'][0] / results['small'][1]
large = results['large'][0] / results['large'][1]
return (small + large) / 2
Add the recovered counts: results['small'][0] + results['large'][0].
Add the totals the same way, then divide once.
def overall_rate(results):
rec = results['small'][0] + results['large'][0]
tot = results['small'][1] + results['large'][1]
return rec / tot
In the kidney-stone data, Treatment A beats B in BOTH stone-size groups, yet B has the higher overall rate. What MUST be true for this to happen?
Simpson's paradox needs an unbalanced confounder: each treatment saw a different share of the harder large-stone group, so the pooled rate weighted them differently and the overall comparison flipped.
Write has_paradox(a, b) that detects Simpson's paradox. a and b are dicts like {'small': (rec, tot), 'large': (rec, tot)}. Return True if treatment a beats b in BOTH groups yet has the LOWER overall (pooled) rate. The helpers _rate and _overall are provided — use them.
def _rate(cell):
return cell[0] / cell[1]
def _overall(treatment):
rec = treatment['small'][0] + treatment['large'][0]
tot = treatment['small'][1] + treatment['large'][1]
return rec / tot
def has_paradox(a, b):
# a beats b in every group, yet loses overall
pass
a beats b in every group when _rate(a[g]) > _rate(b[g]) for both groups.
The paradox is: wins every group AND loses overall, so return wins_groups and not wins_overall.
def _rate(cell):
return cell[0] / cell[1]
def _overall(treatment):
rec = treatment['small'][0] + treatment['large'][0]
tot = treatment['small'][1] + treatment['large'][1]
return rec / tot
def has_paradox(a, b):
a_wins_groups = all(_rate(a[g]) > _rate(b[g]) for g in ('small', 'large'))
a_wins_overall = _overall(a) > _overall(b)
return a_wins_groups and not a_wins_overall
Recap
- Simpson's paradox is a trend in aggregated data that reverses within every subgroup — a real consequence of an unbalanced mix, not an illusion.
- A pooled rate is a weighted average; when the groups are split differently across the things you compare, the weights tilt the answer.
- The cure is stratification: compare within each level of the confounder before you read an overall number.
- The decision risk runs one way — a team that trusts the headline adopts the worse option, every time.
Next you will meet the imbalance that hides an entire class, where even accuracy stops being a number you can trust.