Sampling Methods & Bias
How you choose a sample decides what you can trust — no matter how many people you choose.
Population, sampling frame, sample
Three groups matter, and they are rarely the same group.
The population is everyone (or everything) you want to know about. The sampling frame is the actual list or mechanism you draw from — ideally identical to the population, but often not (a phone directory excludes unlisted numbers; a class roster excludes students who withdrew). The sample is the subset you actually select and measure from that frame.
How you choose the sample from the frame determines what conclusions you can trust. The size of the sample does not fix a bad choice.
Wrong: 'A bigger sample removes bias.'
Right: A larger sample reduces variance — sampling variability, the sample-to-sample wobble — not bias. A biased method aimed at the wrong target just hits that wrong target more precisely. A voluntary-response poll of five million people is still biased; it is just a very precise measurement of the wrong thing.
A simple random sample of size n is chosen so that every possible group of n individuals from the sampling frame has an equal chance of being the sample selected. In practice: number every member of the frame and draw n numbers at random, with no regard to any subgroup.
Wrong: 'An SRS guarantees a representative sample.'
Right: An SRS gives every sample of size n an equal chance, which makes the method unbiased on average, over all the samples it could produce. Any single SRS actually drawn can still be unrepresentative just by chance — for example, it could, by bad luck, pull an unusually old or unusually male-heavy sample. The guarantee is about the long-run behavior of the method, not about any one draw.
Stratified sampling and cluster sampling
Both methods split the population into groups first — but for opposite reasons.
Stratified sampling divides the population into groups (strata) that are internally similar — by grade level, region, or department, for example — and then takes a separate SRS from every stratum. This is done to reduce variance: making sure every important subgroup is represented in its true proportion gives more precise estimates.
Cluster sampling divides the population into groups (clusters) that each look roughly like a small version of the whole population, then randomly selects whole clusters and measures everyone inside the chosen clusters. This is done for cost or convenience — for example, randomly picking 20 classrooms out of a school district and surveying every student in those classrooms, instead of visiting every classroom in the district.
Wrong: 'Stratified and cluster sampling are basically the same.'
Right: Stratified: split into homogeneous groups, then sample from every stratum — used to reduce variance. Cluster: split into groups, then randomly pick whole clusters — used for cost or convenience. The intents are opposite: stratifying guarantees every subgroup appears; clustering deliberately leaves most groups out entirely.
Systematic sampling, convenience sampling, and voluntary response
Systematic sampling picks a random starting point, then selects every k-th individual from the frame (every 10th name on a list, say). It is easy to carry out and, if the list has no hidden pattern lined up with k, behaves much like an SRS.
Convenience sampling selects whichever individuals are easiest to reach (people in the same mall, the same class, the same friend group). Voluntary-response sampling lets individuals choose to participate themselves — a call-in poll, an online rating, a comment section. Both are not probability samples: no individual has a known, controlled chance of selection, because who ends up in the sample is driven by who happens to be reachable, or who happens to feel strongly enough to respond.
Wrong: 'Convenience sampling is fine if the sample is large.'
Right: Convenience and voluntary-response samples are systematically biased regardless of size — the people who are easy to reach, or who feel strongly enough to respond, are rarely a fair stand-in for the population. They are not probability samples, so they have no valid margin of error at all; no formula turns a self-selected crowd into a randomly chosen one.
The bias taxonomy: four ways a sample goes wrong
Even a carefully planned survey can fail in specific, nameable ways:
- Undercoverage: some groups in the population cannot be selected at all, because the sampling frame excludes them (a landline-only phone survey undercovers people who only own cell phones).
- Nonresponse: individuals are selected but do not answer (a mailed survey with a 10% return rate).
- Response bias: people answer, but inaccurately — because of how a question is worded, who is asking, or a wish to give a socially acceptable answer.
- Voluntary-response bias: the sample is made entirely of people who opted themselves in, so only the motivated — often those with strong, one-sided opinions — are represented.
Wrong: 'Nonresponse and undercoverage are the same bias.'
Right: Undercoverage: some groups cannot be selected because the frame excludes them. Nonresponse: selected people do not answer. Response bias: people answer inaccurately. Voluntary response: only the motivated opt in. Naming which one is at fault matters, because the fix differs for each — undercoverage is fixed by building a better frame; nonresponse is fixed by follow-up and incentives; response bias is fixed by rewording questions; voluntary response usually cannot be fixed at all, only avoided by design.
- Frame problem (undercoverage): in 1936, car and telephone ownership skewed toward wealthier households, so the sampling frame already excluded a large share of the voting population before a single ballot was mailed.
- Response problem (voluntary response): recipients chose for themselves whether to mail the ballot back — only about 24% did — so the people who responded were not a random subset of the 10 million contacted, and likely skewed toward those with stronger opinions and more free time.
- Both problems bias WHO ends up in the data, not how precisely that biased group gets measured.
- The sample was enormous (2.4 million responses), yet the poll still called the election wrong, because a bigger sample sharpens the picture of the wrong target; it cannot fix undercoverage or voluntary-response bias.
Wrong: 'Blocking is a survey technique.'
Right: Blocking is the experimental analogue of stratifying in surveys — both group similar units before randomizing, for the same variance-reducing reason. But the vocabulary is not interchangeable: never say 'blocking' for a survey, or 'stratifying' for an experiment. The next lesson, Designing Experiments, covers blocking — along with control, randomization, and replication — in full.
Check your understanding
- Population is who you want to know about; the sampling frame is the actual list you can draw from; the sample is who you actually measure — a frame that leaves people out causes undercoverage before any data is collected.
- A larger sample shrinks variance (sampling variability), not bias — a biased method just measures the wrong target more precisely.
- SRS gives every sample of size n an equal chance (an unbiased method); stratified sampling samples every stratum to cut variance; cluster sampling randomly selects whole groups for convenience; systematic sampling takes every k-th unit.
- Convenience and voluntary-response samples are not probability samples, are biased regardless of size, and have no valid margin of error.
- Undercoverage (excluded by the frame), nonresponse (selected but silent), response bias (answered inaccurately), and voluntary-response bias (self-selected) are four distinct failure modes with four different fixes.
- The 1936 Literary Digest poll shows a huge sample (2.4 million responses) cannot rescue a biased frame and voluntary response — it just measured the wrong group very precisely.