Statistics › 🔬 Confidence & Inference

Hypothesis Testing

How statisticians decide whether a result is real or just the kind of wobble chance produces anyway.

Intro StatisticsAP Statistics levelCollege Stats 101

💡

The big idea: A hypothesis test starts by assuming nothing interesting is going on, then asks a single question: if that assumption were true, how surprising would our data be? If the data would be very unlikely under that assumption, we take that as evidence against it. If the data are the sort of thing that happens all the time by chance, we have no reason to abandon the assumption. That is the whole logic — everything else is bookkeeping.

🎯 By the end, you'll be able to

State a null hypothesis and an alternative hypothesis for a claim
Explain the logic of assuming H0, computing a test statistic, and measuring how surprising the data are
Interpret a p-value honestly as a tail area under the null distribution, and say what it is NOT
Use a significance level alpha to decide between reject and fail-to-reject — and know why we never say accept

📎 You should already know

The normal distribution and z-scores
Sampling distributions and standard error

Start by assuming nothing is happening

Suppose a friend claims a coin is rigged toward heads. You flip it 10 times and get 7 heads. Is that proof? Not really — 7 heads out of 10 happens fairly often with a perfectly fair coin. But 70 heads out of 100 would make you lean forward. Hypothesis testing turns that gut reaction into a rule.

The trick is to argue from the opposite of what you suspect. You begin by assuming the boring, no-effect explanation is true — the coin is fair — and then check whether your data would be strange under that assumption. It is the statistical version of "innocent until proven guilty": the no-effect story gets the benefit of the doubt, and the data have to work to overturn it.

🔑 The two hypotheses

Every test names two competing statements. The null hypothesis \(H_0\) is the boring one — no effect, no difference, nothing going on (the coin is fair, the new method changes nothing). The alternative hypothesis \(H_a\) is the interesting claim you would need evidence to support (the coin is biased, the method changes something). The test never tries to prove \(H_0\); it only asks whether the data give us enough reason to abandon it in favor of \(H_a\).

The logic in three moves

Once the hypotheses are set, every test follows the same three steps:

Assume \(H_0\) is true. This gives us a specific, known distribution that our data summary should follow — the null distribution.
Compute a test statistic. This is a single number that measures how far the data sit from what \(H_0\) expects, in units of standard error. A common form is a z-statistic.
Ask how surprising the data are. Under \(H_0\), where does our statistic fall? Out in a far tail (surprising) or comfortably in the middle (unremarkable)?

Notice we never measure how far the data are from the alternative. Everything is judged against the single, precise world where \(H_0\) holds.

\[ z = \frac{\text{estimate} - \text{value claimed by } H_0}{\text{standard error of the estimate}} \]

A test statistic rescales the gap between what we observed and what H0 predicts into standard-error units, so we can read it against a standard normal null distribution.

How surprising is 'surprising'? Meet the p-value

The p-value puts a number on the surprise. It is the probability, assuming \(H_0\) is true, of getting a test statistic at least as extreme as the one we actually saw. On the null distribution that probability is a tail area: the further out our statistic lands, the smaller the tail beyond it, and the more surprising the data would be under \(H_0\).

In the tool below, the bell is the null distribution — the distribution the statistic would follow if \(H_0\) were true. Slide the observed statistic and watch the shaded tail area, the p-value, shrink as the statistic moves outward. Switch between a two-sided alternative (extreme in either direction) and a one-sided one, and change the significance level to see the rejection region move.

🎮 Interactive: the null distribution and your observed statistic LIVE

The bell is the distribution of the test statistic assuming H0 is true. The shaded tail area is the p-value: the chance, under H0, of a statistic at least as extreme as the one observed. Dashed lines mark the critical values that bound the rejection region for the significance level you pick.

✨ What a p-value is — and is NOT

A p-value is a statement about the data under \(H_0\), not about the hypotheses themselves. A p-value of 0.03 means: if \(H_0\) were true, a result this extreme would occur about 3% of the time. It is not the probability that \(H_0\) is true, and \(1 - p\) is not the probability that \(H_a\) is true. A small p-value says the data would be unusual in the no-effect world; it does not put a probability on which world you are actually in.

Drawing the line: the significance level

To turn a p-value into a yes-or-no decision we pick a threshold in advance: the significance level \(\alpha\), often 0.05, sometimes 0.01. It is the tail area we agree to treat as "too surprising to chalk up to chance." If the p-value falls at or below \(\alpha\), the statistic has landed in the rejection region and we reject \(H_0\). If the p-value is above \(\alpha\), we fail to reject \(H_0\).

Choosing \(\alpha\) is choosing how often we are willing to be fooled: \(\alpha\) is the chance of rejecting a true \(H_0\) just by bad luck. A smaller \(\alpha\) demands stronger evidence before we act.

⚠️ We never 'accept' the null

The two verdicts are reject \(H_0\) and fail to reject \(H_0\) — never "accept \(H_0\)." Failing to reject only means the data were not surprising enough to rule \(H_0\) out; it does not prove \(H_0\) is true. Absence of evidence is not evidence of absence. Likewise, rejecting \(H_0\) supports \(H_a\) but does not prove it — a test weighs evidence, it does not deliver certainty.

📝 Worked example: A bottling line is supposed to fill to a mean of 100 ml. A sample of n = 36 bottles has a mean of 103.5 ml, and the standard error of the mean is 1.667 ml. Test, at alpha = 0.05, whether the line is off target (a two-sided alternative).

State the hypotheses: \(H_0: \mu = 100\) ml (on target) versus \(H_a: \mu \neq 100\) ml (off target, either way).
Assume \(H_0\) and compute the test statistic: \(z = (103.5 - 100) / 1.667 \approx 2.1\). The sample mean sits about 2.1 standard errors above the claimed value.
Find the p-value. Two-sided, so it is both tails: \(p = 2 \times P(Z \geq 2.1) \approx 2 \times 0.0179 = 0.0357\).
Compare to \(\alpha\): \(0.0357 \leq 0.05\), so the statistic falls in the rejection region.

✓ We reject H0 at alpha = 0.05: under the on-target assumption, a mean this far off would happen only about 3.6% of the time, which we agreed to treat as too surprising. We conclude the data are inconsistent with a 100 ml mean — but we have not proven the exact off-target amount, and 'reject' is a weighing of evidence, not a certainty.

Check your understanding

1. In a hypothesis test, the null hypothesis H0 typically represents…

H0 is the boring, no-effect baseline that gets the benefit of the doubt. The interesting claim is the alternative, Ha, which the data must supply evidence for.

2. A two-sided test produces a p-value of 0.03, and you are testing at alpha = 0.05. The correct decision is to…

Since p = 0.03 is at or below alpha = 0.05, the statistic is in the rejection region, so we reject H0. We never 'accept' H0, and a single test does not 'prove' Ha.

3. A p-value of 0.04 is best interpreted as…

A p-value is computed ASSUMING H0 is true — it is a tail area under the null distribution. It is not the probability that H0 (or Ha) is true.

4. A test returns p = 0.20 while alpha = 0.05. The most defensible conclusion is…

Since p = 0.20 is above alpha, we fail to reject H0. That is not proof H0 is true — the data simply were not surprising enough to abandon it. Absence of evidence is not evidence of absence.

✅ Key takeaways

A hypothesis test assumes the null hypothesis H0 (no effect) and asks how surprising the data would be if that were true.
You compute a test statistic — the gap between the estimate and H0's value, in standard-error units — and read it against the null distribution.
The p-value is the tail area under the null distribution: the chance, IF H0 is true, of a statistic at least as extreme as the one observed.
A p-value is not the probability that H0 is true; it only describes how unusual the data would be under H0.
Pick a significance level alpha in advance; reject H0 when p is at or below alpha, otherwise fail to reject.
The verdicts are reject and fail-to-reject — never accept — because failing to reject does not prove H0, and rejecting does not prove Ha.