Type I and Type II Errors and Power
Every hypothesis test can be wrong in two different ways — and one number, power, measures how often you catch a real effect.
Two ways to be wrong
Think of a smoke detector. It faces a genuinely uncertain world and has to make a call, so it can fail in two completely different ways. It can shriek when you are only making toast — a false alarm. Or it can stay silent during a real fire — a miss. Making it more sensitive cuts down the misses but causes more false alarms; making it less twitchy does the reverse. You cannot drive both kinds of mistake to zero at once.
A hypothesis test is exactly this kind of detector. It starts from a null hypothesis \(H_0\) (the "nothing unusual is happening" claim) and decides whether the data give enough reason to reject it. Because the data are noisy, the test can be wrong in two opposite ways — the statistical versions of the false alarm and the miss.
The four outcomes of a test
Reality is either that \(H_0\) is true or that it is false, and your test either rejects \(H_0\) or does not. That makes four combinations — two right, two wrong:
- \(H_0\) true, you do not reject — correct. No effect, and you said so.
- \(H_0\) false, you reject — correct, and this is the outcome you usually want: a real effect, detected.
- \(H_0\) true, but you reject it — a Type I error: a false alarm. You announced an effect that is not there.
- \(H_0\) false, but you fail to reject it — a Type II error: a miss. A real effect was there and you walked past it.
Notice the errors are not symmetric in meaning: one invents an effect, the other overlooks one.
Two curves, one cutoff
The cleanest way to see all of this is to draw the sampling distribution of your test statistic under both stories at once. The left curve is what the statistic looks like if \(H_0\) is true; the right curve is what it looks like if a specific alternative is true instead. Because data are noisy, the two curves overlap — and that overlap is where errors live.
You reject \(H_0\) whenever the statistic lands past a cutoff. That single line splits both curves into shaded pieces: alpha is the sliver of the null curve past the cutoff (a true null wrongly rejected), beta is the chunk of the alternative curve on the not-reject side (a real effect missed), and power is the rest of the alternative curve, past the cutoff. Change alpha to slide the cutoff, or raise the effect size to pull the curves apart, and watch the three areas respond.
What makes a test powerful
Since power falls out of the overlap between the two curves, anything that reduces that overlap raises power. Three levers do it:
- A bigger true effect. The farther the real value sits from the null, the more the alternative curve slides to the right, clearing the cutoff. Big effects are easy to detect; tiny ones are not.
- A larger sample size \(n\). More data shrinks the standard error \(\sigma/\sqrt{n}\), making both curves narrower and taller so they overlap less. This is the lever you actually control — plan for enough \(n\).
- A larger alpha. Loosening the cutoff (0.05 instead of 0.01) raises power, but only by accepting more Type I errors. It trades one error for the other rather than removing overlap.
Lower variability in the data (or a better-designed study) helps too, for the same reason a bigger \(n\) does: it narrows the curves.
Alpha and beta are long-run error rates of the procedure, not statements about any single result. A test that rejects at the 0.05 level does not mean there is a 95% chance the effect is real, and a p-value is not the probability that \(H_0\) is true — it is how surprising your data would be if \(H_0\) were true.
Just as important: failing to reject \(H_0\) is not proof that \(H_0\) is true. A non-significant result from a low-power test (small effect, small sample) is often just a miss — absence of evidence, not evidence of absence. This is why power matters before you run the study, not only after.
- Find the standard error: SE = sigma / sqrt(n) = 15 / sqrt(25) = 15 / 5 = 3.
- Find the cutoff. A one-sided test at alpha = 0.05 rejects when the sample mean exceeds mu_0 + 1.645 times SE = 100 + 1.645 times 3 = 104.9 (approximately).
- Assume the alternative is the truth (mean = 106) and ask how often the sample mean still falls below the cutoff — that is beta: \(\beta = P(\bar{x} < 104.9 \mid \mu = 106) = P\!\left(Z < \tfrac{104.9 - 106}{3}\right) = P(Z < -0.37) \approx 0.36.\)
- Power is the complement: power = 1 minus beta ≈ 1 − 0.36 = 0.64.
- Now raise the sample size to n = 100. Then SE = 1.5, the cutoff is 100 + 1.645 times 1.5 ≈ 102.5, and \(\beta = P(Z < \tfrac{102.5 - 106}{1.5}) = P(Z < -2.35) \approx 0.01\), so power jumps to about 0.99.
Check your understanding
- A hypothesis test can be wrong two ways: a Type I error (rejecting a true H0 — a false alarm, rate alpha) and a Type II error (failing to reject a false H0 — a miss, rate beta).
- You set alpha directly as the significance level; beta is left over and depends on the effect size, the sample size, and alpha.
- Power = 1 minus beta is the probability of detecting a real effect. Higher power means fewer misses.
- Alpha and beta trade off: with effect size and n fixed, lowering alpha raises beta (and lowers power), and vice versa.
- You raise power by studying a bigger true effect, collecting a larger sample, or accepting a larger alpha — and a bigger sample is the lever you truly control.
- Be honest: alpha and beta are long-run error rates, a p-value is not the probability H0 is true, and failing to reject H0 is not proof that H0 is true.