Statistics › 🔬 Confidence & Inference

P-values & Significance

What a p-value actually measures — and the tempting misreadings to avoid.

Intro StatisticsAP Statistics level

💡

The big idea: A p-value answers one narrow question: if the null hypothesis were true, how often would pure chance hand you a result at least as extreme as the one you actually got? A small p-value means your data would be surprising in a world with no real effect — so you start to doubt that world. It is a measure of surprise under the null, nothing more, and it is emphatically not the probability that the null is true.

🎯 By the end, you'll be able to

State the precise definition of a p-value as a conditional tail probability under H0
Read a small p-value as 'the data are surprising if H0 is true', and compare it to a significance level alpha
Tell a one-sided from a two-sided test and know why the two-sided p-value is doubled
Avoid the classic misreadings: a p-value is not the probability H0 is true, and 'not significant' is not 'no effect proven'

📎 You should already know

The normal distribution and z-scores
Sampling distributions and standard error
Null and alternative hypotheses (H0 vs H1)

Start with a world where nothing is going on

Every significance test begins by playing devil's advocate. You assume the null hypothesis \(H_0\) — the boring, no-effect explanation: the coin is fair, the two groups are really the same, the new method changes nothing. Then you ask a single question:

If that dull world were true, how often would ordinary chance alone produce data at least as extreme as what I just saw?

That 'how often' is the p-value. If the answer is 'all the time', your data fit the no-effect world fine. If the answer is 'almost never', either something unusual happened or the no-effect world is wrong — and the rarer the coincidence, the more you lean toward the second reading.

🔑 The p-value, precisely

A p-value is the probability, computed assuming \(H_0\) is true, of getting a test statistic at least as extreme as the one actually observed. Small p = the data would be surprising under \(H_0\). It is a statement about the data given the null — not a statement about how likely the null is.

Turning data into a tail area

To make this concrete we standardize. Under \(H_0\), many test statistics behave like a standard normal curve centered at 0 — the null distribution. Your data collapse to a single number, the observed statistic \(z_{\text{obs}}\): how many standard errors your result sits from the null value.

The p-value is then just an area in the tail of that curve — the shaded slice of outcomes as extreme as, or more extreme than, yours. Drag the observed statistic in the tool below and watch the tail area (the p-value) change. Switch between a one-sided and a two-sided alternative, and change the significance level \(\alpha\) to see when the shaded p-tail slips past the critical value.

🎮 Interactive: the p-value as a shaded tail LIVE

The bell is the null distribution (what H0 predicts). The blue line is your observed statistic; the shaded red area beyond it is the p-value. Two-sided shades both tails. The dashed line marks the critical value for alpha: reject H0 exactly when the observed statistic falls past it — equivalently, when the shaded p is at most alpha.

\[ p_{\text{one-sided}} = P\!\left(Z \ge z_{\text{obs}}\right) \qquad p_{\text{two-sided}} = P\!\left(|Z| \ge |z_{\text{obs}}|\right) = 2\,P\!\left(Z \ge |z_{\text{obs}}|\right) \]

For a right-tailed alternative you take one tail; when the direction is not specified in advance you count both tails, which doubles the area.

One-sided or two-sided?

The choice depends on the question you set before looking at the data:

A one-sided test asks about a specific direction ('is the new time faster?'). Only one tail counts as extreme.
A two-sided test asks whether things differ at all, in either direction. Both tails count, so the p-value is twice as large for the same \(z_{\text{obs}}\).

Because two-sided is more conservative and does not require guessing the direction, it is the common default. Picking one-sided after seeing which way the data went is a way to fool yourself — decide up front.

⚠️ What a p-value is NOT

It is dangerously easy to over-read a p-value. Guard against these:

It is not the probability that \(H_0\) is true. \(H_0\) is either true or not; the p-value is computed assuming it is true, so it cannot also be its probability.
A p-value of 0.03 does not mean '3% chance the result is a fluke and 97% chance the effect is real'.
It does not measure the size or importance of an effect — only how surprising the data are under the null. A tiny, unimportant effect can give a small p-value with enough data.

Comparing to a threshold

To turn the p-value into a yes/no decision, you fix a significance level \(\alpha\) in advance — often 0.05 — as the line for 'surprising enough'. If \(p \le \alpha\) you reject \(H_0\) and call the result statistically significant; if \(p > \alpha\) you do not reject it. The \(\alpha\) you pick is also your tolerated rate of wrongly rejecting a true \(H_0\) — the false-alarm rate.

✨ 'Not significant' is not 'no effect'

Failing to reject \(H_0\) means the data did not give enough evidence against it at your chosen \(\alpha\) — not that \(H_0\) has been shown true. Absence of evidence is not evidence of absence: a real effect can hide behind a large p-value simply because the sample was too small to detect it. 'Do not reject' is an honest shrug, not a proof of no effect.

📝 Worked example: A two-sided test produces a standardized statistic of z = 2.10. Using alpha = 0.05, find the p-value and state the decision — carefully.

Find one tail. From the standard normal, \(P(Z \ge 2.10) \approx 0.0179\).
It is two-sided, so count both tails: \(p = 2 \times 0.0179 \approx 0.036\).
Compare to \(\alpha = 0.05\). Since \(0.036 \le 0.05\), the result falls in the rejection region.
Interpret honestly: a no-effect world would produce a result this extreme only about 3.6% of the time, so the data cast doubt on \(H_0\).

✓ p ≈ 0.036, which is at most 0.05, so we reject H0 at the 5% level. This says the data are surprising under H0 — it does NOT say there is a 3.6% chance H0 is true.

Check your understanding

1. Which is the correct definition of a p-value?

A p-value is a conditional tail probability: it assumes H0 is true and asks how often chance alone would give data this extreme or more so.

2. A study reports p = 0.03. Which statement is correct?

The p-value is P(data this extreme | H0 true) = 0.03. It says nothing directly about the probability of H0, and a hypothesis test never proves a hypothesis.

3. A two-sided test gives z = 2.50 with a p-value of about 0.012. If instead you had (in advance) run a one-sided test in the observed direction, the p-value would be about…

A two-sided p counts both tails; a one-sided p in the observed direction counts just one, so it is half: about 0.006.

4. A test gives p = 0.20 with alpha = 0.05, so you do not reject H0. The best interpretation is…

Failing to reject means insufficient evidence against H0 at this alpha, not that H0 is true. Absence of evidence is not evidence of absence — the effect may simply be too small to detect here.

✅ Key takeaways

A p-value is the probability, assuming H0 is true, of a result at least as extreme as the one observed — a measure of surprise under the null.
Small p = the data would be unusual in a no-effect world, so you doubt that world; compare p to a preset significance level alpha and reject H0 when p ≤ alpha.
A p-value is NOT the probability that H0 is true, and it does not measure the size or importance of an effect.
A two-sided test counts both tails (so its p-value is double the one-sided value); choose the sidedness before seeing the data.
'Not significant' means not enough evidence against H0 — not proof that there is no effect.