Statistics › 🧪 Collecting Data

Designing Experiments

Why deliberately assigning treatments, instead of just watching, is what finally lets you say one thing caused another.

Intro StatisticsCollege intro statistics level

Illustrative image (AI-generated).

💡

The big idea: An experiment differs from an observational study in one crucial way: the researcher deliberately assigns treatments instead of recording what subjects already do. Four principles — control, randomization, replication, and blocking — turn that intervention into a comparison you can trust. But even a flawless experiment only ever answers one of two separate questions: whether the treatment caused the effect, and whether the result applies beyond the subjects actually studied. This lesson builds the vocabulary to keep those two questions straight.

🎯 By the end, you'll be able to

State the four principles of experimental design — control, randomization, replication, and blocking — and explain what each one buys you
Distinguish a control group, a placebo, and blinding as three separate tools, and correctly pair single-blind and double-blind with what each one hides
Choose among a completely randomized design, a randomized block design, and a matched-pairs design for a given experimental scenario
Explain how blocking in an experiment plays the same role as stratifying in a survey, while remaining a different procedure at a different stage of data collection
Apply the scope-of-inference rule to state whether a study's result can be called causal, generalizable, both, or neither

📎 You should already know

Observational Studies vs Experiments
Sampling Methods & Bias
The basic logic of a hypothesis test

From observing to intervening

An observational study watches subjects and records what happens; it can uncover an association, but it can never rule out some lurking variable as the real explanation. An experiment is different: the researcher deliberately imposes treatments on subjects and controls who gets what. That act of imposing and randomizing treatment is what eventually licenses a causal claim — the subject of this lesson.

Good experiments do not happen by accident. They are built on four design principles that work together to rule out competing explanations for what you observe.

The four principles

Every well-designed experiment leans on four principles:

Control — compare the treatment against a baseline group, so you have something to measure the effect against.
Randomization — assign subjects to treatments by a chance mechanism (a coin flip, a random number generator), so the groups start out alike, on average, in every way except the treatment.
Replication — apply each treatment to more than one subject, so ordinary chance variation between individuals doesn't get mistaken for a treatment effect.
Blocking — first group similar subjects together, then randomize treatments within each group, so a known source of variability doesn't drown out the effect you're looking for.

Control and randomization are what make a causal claim possible at all; replication and blocking are what make the resulting comparison precise.

⚠️ A control group, a placebo, and blinding are three different tools

A common mix-up is to say 'the placebo group is the control' as if the two words meant the same thing. They don't.

A control group is whatever provides the baseline for comparison — subjects who get no treatment, the current standard treatment, or a placebo. Control is about the comparison.
A placebo is a fake treatment, made to look identical to the real one, given so subjects don't know whether they received the active treatment. Placebo is about disguising which treatment a subject got.
Blinding is hiding who got which treatment — from the subjects alone (single-blind), or from both the subjects and the people evaluating outcomes (double-blind) — so expectations don't bias the results.

These are three distinct tools. A control group need not be a placebo group (it might get the existing standard treatment instead). An experiment can use a placebo without being blinded, if the evaluator can still tell the groups apart. And you can blind an experiment that compares two active treatments, with no placebo anywhere in it.

Completely randomized, randomized block, and matched pairs

Randomization can be organized in different ways depending on what you already know about your subjects before you start:

Completely randomized design — every subject is randomly assigned to a treatment group from the whole pool at once, with no preliminary grouping. The simplest option; use it when subjects are reasonably similar to begin with.
Randomized block design — subjects are first sorted into blocks of similar units (by sex, by age group, by field location), and randomization happens separately within each block. This holds a known nuisance variable steady so it can't masquerade as a treatment effect.
Matched-pairs design — a special case of blocking where each block has exactly two units. Either the same subject is measured under both treatments (before/after, or the two treatments in random order), or two similar subjects are matched into a pair and the treatments are assigned randomly within that pair.

✨ Blocking is the experimental analogue of stratifying — not the same thing

Blocking in an experiment and stratifying in a survey share the same instinct — split subjects into similar groups before letting chance act — but they operate at two different stages of data collection.

Stratifying (from Sampling Methods & Bias) happens when you select a sample: you divide the population into strata and sample from each, so your sample mirrors the population's makeup.
Blocking happens when you assign treatments in an experiment you're already running: you divide your subjects into blocks and randomize the treatment within each block, so a known difference between blocks doesn't get confused with a treatment effect.

Same idea — group like with like — applied at different steps: stratifying controls who gets sampled, blocking controls who gets which treatment. Don't use the two words interchangeably.

Matched pairs, and its own-control trick

The matched-pairs design deserves a closer look, because it reappears later in the course. Two variations both go by the name 'matched pairs':

Each subject as their own control — the same subject receives both treatments, or is measured before and after, typically in a randomly chosen order. Comparing a person to themselves cancels out all the ways people differ from one another.
Matched similar pairs — two different subjects who are alike on variables that matter (twins, litter-mates, plots with the same soil) are paired up, and the treatments are then randomly assigned within each pair — one member gets treatment A, the other treatment B.

Either way, the analysis works on the differences within each pair, not on the two groups as if they were independent. That is exactly what a later lesson calls the paired t procedure — matched pairs is the design; paired t is the analysis built for it.

📝 Worked example: An agronomist has 12 plots of land to test a new fertilizer against the standard fertilizer. The 12 plots sit across 3 fields, and the fields are known to differ in soil quality. She wants a design that won't let the soil differences masquerade as a fertilizer effect.

The fields are a known source of variability, and there are more than 2 of them, so this is not a matched-pairs situation.
Treat each field as a block: 3 blocks of 4 plots each.
Within each block, randomly assign 2 plots to the new fertilizer and 2 to the standard fertilizer.
Every block now contributes one clean comparison of new versus standard, with soil quality held roughly constant inside each comparison.

✓ Use a randomized block design, blocking by field: randomize the fertilizer assignment separately within each of the 3 fields. This uses all four principles together — control (standard fertilizer as the baseline), randomization within each block, replication (multiple plots per treatment), and blocking (by field) — so soil-quality differences between fields cannot be mistaken for a fertilizer effect.

The capstone question: what can you actually conclude?

Every study answers two independent questions, and mixing them up is the single most common conceptual error in experimental design:

Were subjects randomly assigned to treatments?
Were subjects randomly sampled from a larger population?

These are separate design choices, and each licenses a separate kind of conclusion:

	Random sampling	No random sampling
Random assignment	Causal claim, generalizes to population	Causal claim, only about these subjects
No random assignment	Association, generalizes to population	Association, only about these subjects

🔑 Two different jobs — don't swap them

Random assignment is what licenses a causal claim: because chance, not the subjects' own choices or traits, decided who got which treatment, a resulting difference between groups can be attributed to the treatment.

Random sampling is what licenses generalizing the result to a wider population: because chance decided who ended up in the study, the study's subjects represent that population.

These two jobs are orthogonal — one has nothing to do with the other. A perfectly randomized experiment on a convenience sample of volunteers still supports a causal claim; it just can't be generalized past those volunteers. Do not say 'random assignment lets us generalize to the whole population' — that sentence assigns randomization's job to sampling's job.

Check your understanding

1. In a drug trial, each treatment is given to 30 subjects instead of just one or two. Which design principle does this serve most directly?

Applying each treatment to many subjects — replication — means ordinary person-to-person variation averages out, so it isn't mistaken for a treatment effect. Randomization is about how subjects are assigned, not how many receive each treatment.

2. A study gives one group the new drug and another group a sugar pill made to look identical, and neither the subjects nor the technician recording outcomes knows who received which pill. Which statement is correct?

The placebo group provides the comparison baseline (control), the placebo itself is the fake treatment, and hiding assignment from both subjects and the evaluator makes this double-blind — three distinct, simultaneously-present tools.

3. A survey researcher divides the population into age strata before sampling; an experimenter divides subjects into blocks by age before randomizing treatment. Which statement is correct?

Stratifying happens at the sampling stage (who is selected); blocking happens at the assignment stage of an experiment (who gets which treatment). The instinct is the same; the procedure and the stage are different.

4. Researchers randomly assign volunteers, who signed up online rather than being randomly sampled, to a treatment group or a control group, and find a significant difference. What can they conclude?

Random assignment licenses the causal claim regardless of how the subjects were recruited. Because there was no random sampling, that causal claim cannot be generalized beyond the volunteers actually studied.

✅ Key takeaways

Good experiments rest on four principles: control (a baseline to compare against), randomization (chance decides who gets which treatment), replication (each treatment reaches multiple subjects), and blocking (grouping similar subjects before randomizing within each group).
A control group, a placebo, and blinding are three distinct tools: control provides the comparison baseline, a placebo is a fake treatment, and blinding hides who received which treatment (single-blind: from subjects; double-blind: from subjects and evaluators too).
A completely randomized design randomizes the whole pool at once; a randomized block design randomizes separately within groups of similar subjects; matched pairs is blocking taken to its limit — blocks of exactly two.
Blocking (experiments) and stratifying (surveys) share the instinct of grouping like with like, but blocking controls treatment assignment while stratifying controls sample selection.
Random assignment licenses a causal claim; random sampling licenses generalizing to a population. The two are independent design choices — random assignment alone does not earn you generalization.
Matched-pairs data — the same subject measured twice, or matched similar subjects — is analyzed later in the course with the paired t procedure, which works on the within-pair differences.