Comparing Two Proportions
One shared standard-error trick for the test, and a different one for the interval.
Illustrative image (AI-generated).
💡
The big idea: Comparing two groups — a treatment versus a control, campaign A versus campaign B — means comparing two sample proportions, \( \hat{p}_1 \) and \( \hat{p}_2 \). The parameter of interest is the difference \( p_1 - p_2 \). As with a single proportion, the test and the interval both need a standard error for that difference, but they build it two different ways: the test pools the two samples into one shared estimate, because its null hypothesis claims the two proportions are equal; the interval keeps the two samples separate, because it is estimating a difference that may not be zero. Pooled-versus-unpooled is the trap this lesson exists to close.
From one proportion to a difference of two
The one-proportion lesson compared a single \( \hat{p} \) to a claimed value \( p_0 \). Now the question is whether two groups have the same underlying proportion — a new drug versus a placebo, campaign A versus campaign B. Each group gives its own sample proportion, \( \hat{p}_1 \) and \( \hat{p}_2 \), and the parameter we care about is the difference \( p_1 - p_2 \).
Both the test and the interval need a standard error for \( \hat{p}_1 - \hat{p}_2 \). They are built differently, and mixing them up is the most common mistake in this topic.
🔑
Pooling is justified only under H0
The test's null hypothesis is \( H_0: p_1 = p_2 \) — it claims the two groups share one true proportion. If that is true, the best estimate of that shared proportion uses all the data from both groups combined: the pooled proportion. That is why the test's standard error is built from the pooled value.
The interval is a different question: it estimates \( p_1 - p_2 \), which is allowed to be anything, including nonzero. There is no shared value to assume, so pooling would be estimating a common proportion that the interval does not believe exists. The interval keeps each group's own \( \hat{p}_1 \) and \( \hat{p}_2 \) — unpooled.
\[ \hat{p}_{\text{pool}} = \frac{x_1 + x_2}{n_1 + n_2} \]
The pooled proportion: total successes from both groups combined, divided by the total sample size — the single shared proportion that H0 claims exists.
\[ z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}_{\text{pool}}(1-\hat{p}_{\text{pool}})\left(\dfrac{1}{n_1}+\dfrac{1}{n_2}\right)}} \]
The two-proportion test statistic: the observed gap divided by a standard error built from the pooled proportion, because H0 assumes p1 = p2.
Conditions for the test — checked per group
Check the Normal approximation separately for each group, not once overall:
- \( n_1\hat{p}_1 \geq 10 \) and \( n_1(1-\hat{p}_1) \geq 10 \) — group 1, using this course's \( \geq 10 \) convention.
- \( n_2\hat{p}_2 \geq 10 \) and \( n_2(1-\hat{p}_2) \geq 10 \) — group 2.
- Both samples random and independent of each other (e.g. two separate random samples, or random assignment to two groups).
- Each sample no more than 10% of its own population.
Four numbers, four checks — one group passing does not tell you anything about the other.
\[ (\hat{p}_1 - \hat{p}_2)\;\pm\; z^{*}\sqrt{\dfrac{\hat{p}_1(1-\hat{p}_1)}{n_1}+\dfrac{\hat{p}_2(1-\hat{p}_2)}{n_2}} \]
The confidence interval for p1 minus p2: the observed gap, plus-or-minus a critical value times an unpooled standard error — each group keeps its own p-hat.
Why the interval cannot reuse the pooled SE
Pooling only makes sense if you are willing to assume \( p_1 = p_2 \). The interval's entire purpose is to say how big \( p_1 - p_2 \) plausibly is — including the possibility that it is not zero. Using the pooled proportion there would quietly smuggle in the test's assumption, the one the interval is supposed to be agnostic about. So the interval uses each group's own \( \hat{p}_1(1-\hat{p}_1)/n_1 \) and \( \hat{p}_2(1-\hat{p}_2)/n_2 \), added under the square root. The same per-group conditions from above apply here too.
📝 Worked example: Campaign A converts 45 of 100 visitors (\( \hat{p}_1 = 0.45 \)). Campaign B converts 30 of 120 visitors (\( \hat{p}_2 = 0.25 \)). Test \( H_0: p_1 = p_2 \) versus \( H_a: p_1 \neq p_2 \) at \( \alpha = 0.05 \), and separately build a 95% confidence interval for \( p_1 - p_2 \).
- Conditions, per group: n1 phat1 = 45, n1(1-phat1) = 55; n2 phat2 = 30, n2(1-phat2) = 90 — all >= 10. Both samples random and independent, each under 10% of its population. Proceed.
- Pooled proportion (for the test): phat_pool = (45+30)/(100+120) = 75/220 = 0.3409.
- Pooled SE = sqrt( phat_pool(1-phat_pool) x (1/n1 + 1/n2) ) = sqrt( 0.3409 x 0.6591 x (0.01 + 0.008333) ) = sqrt(0.2247 x 0.018333) = sqrt(0.004119) = 0.06418.
- Test statistic: z = (phat1 - phat2)/pooled SE = (0.45 - 0.25)/0.06418 = 0.20/0.06418 = 3.116.
- Two-sided p-value for z = 3.116 is about 0.0018 — far below alpha = 0.05, so reject H0: evidence that p1 != p2.
- Unpooled SE (for the interval): sqrt( phat1(1-phat1)/n1 + phat2(1-phat2)/n2 ) = sqrt(0.2475/100 + 0.1875/120) = sqrt(0.002475 + 0.0015625) = sqrt(0.0040375) = 0.06354.
- 95% CI: (phat1 - phat2) +/- 1.96 x 0.06354 = 0.20 +/- 0.1245, i.e. about (0.0755, 0.3245).
- Note the two standard errors are different numbers: 0.06418 (pooled, for the test) versus 0.06354 (unpooled, for the interval). Using the wrong one shifts both the z-statistic and the interval.
✓ z ~ 3.116, two-sided p-value ~ 0.0018 < 0.05, so we reject H0: the data give evidence campaign A and campaign B have different conversion proportions. Separately, the 95% confidence interval for p1 - p2 is about (0.076, 0.325), built from the unpooled SE 0.06354 — a different number from the test's pooled SE, 0.06418, not an interchangeable one.
⚠️
Do not swap pooled and unpooled
WRONG: using the unpooled standard error, \( \sqrt{\hat{p}_1(1-\hat{p}_1)/n_1 + \hat{p}_2(1-\hat{p}_2)/n_2} \), inside the test statistic. RIGHT: pool, because \( H_0 \) claims \( p_1 = p_2 \), so both samples are estimating the same shared proportion.
WRONG (the mirror-image mistake): using the pooled standard error inside the confidence interval. RIGHT: unpooled, because the interval assumes no equality between the groups — pooling would bake in the very claim the interval is meant to be neutral about.
WRONG (the mirror-image mistake): using the pooled standard error inside the confidence interval. RIGHT: unpooled, because the interval assumes no equality between the groups — pooling would bake in the very claim the interval is meant to be neutral about.
⚠️
Evidence is not proof, and significant is not the same as caused
WRONG: \'the test proves campaigns A and B differ.\' RIGHT: the test gives evidence that \( p_1 \neq p_2 \); it does not prove it, and statistical significance is not the same as a practically important difference — with a large enough \( n_1 \) and \( n_2 \), even a tiny, unimportant gap can produce a small p-value.
WRONG: \'a significant difference between two observed groups means the treatment caused it.\' RIGHT: a causal claim requires random assignment of subjects to the two groups. Without it — two groups that simply exist, like visitors who happened to see campaign A versus campaign B — a significant difference is an association, not proof of cause. See the observational-studies-versus-experiments lesson for how random assignment changes what you're allowed to conclude.
WRONG: \'a significant difference between two observed groups means the treatment caused it.\' RIGHT: a causal claim requires random assignment of subjects to the two groups. Without it — two groups that simply exist, like visitors who happened to see campaign A versus campaign B — a significant difference is an association, not proof of cause. See the observational-studies-versus-experiments lesson for how random assignment changes what you're allowed to conclude.
⚠️
Check both groups, not the data once overall
WRONG: checking the success/failure condition a single time on the combined or pooled data. RIGHT: check \( n_1\hat{p}_1 \geq 10 \), \( n_1(1-\hat{p}_1) \geq 10 \), \( n_2\hat{p}_2 \geq 10 \), and \( n_2(1-\hat{p}_2) \geq 10 \) — four separate numbers. Also confirm both samples are random and independent of one another, and each is at most 10% of its own population.
🎮 Interactive: the null distribution and the p-value LIVE
This widget draws the generic standard Normal null distribution that any standardized z-statistic is compared against. Slide the observed z to see the shaded tail area (the p-value) and the decision at your chosen alpha. It does not compute phat1, phat2, or the pooled/unpooled standard errors for you — you supply z from the pooled-SE formula worked through in this lesson.
Check your understanding
1. Testing \( H_0: p_1 = p_2 \), which standard error belongs in the test statistic?
H0 claims one shared proportion, so the samples are pooled to estimate it, and that pooled value goes into the standard error.
2. Building a 95% confidence interval for p1 - p2, which standard error is correct?
The interval does not assume p1 = p2, so there is no shared proportion to pool — each group keeps its own p-hat in its own term.
3. A two-proportion test gives z = 3.12, p-value = 0.0018, at alpha = 0.05, comparing two campaigns the visitors were not randomly assigned to. What's the correctly worded conclusion?
The small p-value supports rejecting H0, but without random assignment the result is evidence of association, not causation, and it is never a probability statement about p1 or p2.
4. What conditions must hold before trusting the Normal approximation in a two-proportion test?
Each group needs its own success/failure check against its own p-hat — four separate numbers — plus randomness, independence between groups, and the 10% condition for each sample.
✅ Key takeaways
- Pool the two samples' proportions for the hypothesis test, because H0: p1 = p2 assumes one shared proportion.
- Keep the two sample proportions separate (unpooled) for the confidence interval, since it estimates a difference that may not be zero.
- Check the success/failure conditions separately for each group, against each group's own p-hat, using the >= 10 convention.
- A small p-value is evidence the two proportions differ, not proof — and a difference is only a caused effect if the groups were randomly assigned.
- Statistical significance depends on sample size; a 'significant' gap can still be too small to matter in practice.