A single decision tree is a nervous learner. Train it twice on two slightly different samples of the same data and you can get two completely different trees, because every split depends on exactly which rows it happened to see. That fragility is called variance, and it is the weakness this lesson fixes. The remedy is not a cleverer tree; it is many trees. An ensemble combines dozens of weak, wobbly models into one that is far more stable and accurate than any member. The two ways to combine them — bagging and boosting — are the workhorses behind random forests and gradient boosting, the models that win most structured-data competitions. They sound similar and behave oppositely, so the rest of this lesson is about keeping them straight.
flowchart TD D["original data"] --> B1["bootstrap sample 1"] D --> B2["bootstrap sample 2"] D --> B3["bootstrap sample 3"] B1 --> T1["tree 1"] B2 --> T2["tree 2"] B3 --> T3["tree 3"] T1 --> V["average or vote"] T2 --> V T3 --> V style V fill:#166534,color:#fff
Bagging: a committee that votes
Bagging is short for bootstrap aggregating, and both words matter. Bootstrap means sampling the training data with replacement: you draw a new dataset the same size as the original, so some rows appear several times and others drop out entirely. Each tree is trained on a different bootstrap sample, so the trees see slightly different worlds and make different mistakes. Aggregating means combining their predictions — average them for a number, or take a majority vote for a class. Because the trees made independent errors, those errors cancel when you combine them. That cancellation is the whole point: it drives variance down without making the individual trees any cleverer. A random forest is bagging plus one extra twist, where each split only considers a random subset of features, which makes the trees disagree even more.
rows = ['a', 'b', 'c', 'd', 'e']
# three bootstrap samples, each the same size as the data,
# drawn with replacement so rows can repeat or be missing
samples = [[0, 2, 2, 3, 0], [1, 1, 4, 0, 2], [3, 3, 0, 1, 4]]
for i, idx in enumerate(samples, 1):
drawn = [rows[j] for j in idx]
missing = [r for r in rows if r not in drawn]
print('sample', i, 'drawn:', drawn, '| missing:', missing)
Look at what resampling produced. Every sample is the same size as the original, yet rows repeat and others vanish — sample one lost two rows entirely and read one row twice. Hand those three samples to three trees and you get three models that agree on the broad shape but disagree on the details, which is exactly what you want. Averaging models that disagree is where the variance reduction comes from; averaging models that always said the same thing would cancel nothing. This is why a random forest forces extra disagreement with random feature subsets — diversity between the trees is the ingredient that makes the average sharp.
def stump_predict(stump, x):
thr, left, right = stump
return left if x <= thr else right
# three shallow trees, as if each were trained on a different bootstrap sample
s1 = (3.0, 2.0, 6.0)
s2 = (4.5, 3.0, 7.0)
s3 = (5.5, 4.0, 8.0)
stumps = [s1, s2, s3]
xs = [1, 2, 3, 4, 5, 6, 7, 8]
ys = [2, 4, 5, 4, 5, 7, 9, 8]
def mse(preds):
return sum((y - p) ** 2 for y, p in zip(ys, preds)) / len(ys)
for i, s in enumerate(stumps, 1):
preds = [stump_predict(s, x) for x in xs]
print('stump', i, 'MSE:', round(mse(preds), 2))
bag = [sum(stump_predict(s, x) for s in stumps) / len(stumps) for x in xs]
print('bagged MSE:', round(mse(bag), 2))
Why averaging helps
Each tree alone is a mediocre fit — one misses badly, another is decent, a third is best. But the bagged prediction, the simple average of the three, beats the typical individual tree. No single model dominates every point; each is wrong in its own way, and those uncorrelated mistakes wash out in the mean. Notice what bagging did not do: it never made any tree smarter. It left the bias — the error a tree makes even with infinite data — untouched, and instead attacked the variance, the part of the error that comes from latching onto one particular sample. Bagging is the right tool when your models are flexible but jittery, which describes a deep decision tree perfectly.
flowchart LR T1["tree 1 fits the data"] --> R1["residuals: what it got wrong"] R1 --> T2["tree 2 fits the residuals"] T2 --> R2["smaller residuals"] R2 --> T3["tree 3 fits the new residuals"] style T1 fill:#3776ab,color:#fff style T3 fill:#166534,color:#fff
Boosting: a team that studies its mistakes
Boosting flips the strategy. Instead of training many independent trees and averaging them, you train trees one after another, and each new tree focuses on the examples the running ensemble still gets wrong. The mechanism is the residual: after a tree predicts, you subtract its predictions from the true targets, and whatever is left over becomes the target for the next tree. Add the new tree's predictions to the running total and the error shrinks. Where bagging lowers variance by averaging look-alike models, boosting lowers bias by stacking shallow weak learners into one expressive model. A single stump is almost useless; dozens of them, each correcting the last, can fit a deeply non-linear function.
def stump_fit(xs, ys):
pairs = sorted(zip(xs, ys))
n = len(pairs)
best = None
best_sse = None
for i in range(n - 1):
if pairs[i][0] == pairs[i + 1][0]:
continue
thr = (pairs[i][0] + pairs[i + 1][0]) / 2
left = [pairs[j][1] for j in range(i + 1)]
right = [pairs[j][1] for j in range(i + 1, n)]
ml = sum(left) / len(left)
mr = sum(right) / len(right)
sse = sum((v - ml) ** 2 for v in left) + sum((v - mr) ** 2 for v in right)
if best_sse is None or sse < best_sse:
best_sse = sse
best = (thr, ml, mr)
return best
def stump_predict(stump, x):
thr, left, right = stump
return left if x <= thr else right
xs = [1, 2, 3, 4, 5, 6, 7, 8]
ys = [2, 4, 5, 4, 5, 7, 9, 8]
lr = 0.5
# round 1: a stump fits the targets
s1 = stump_fit(xs, ys)
pred1 = [stump_predict(s1, x) for x in xs]
# round 2: a stump fits the RESIDUALS the first stump left behind
residual = [y - p for y, p in zip(ys, pred1)]
s2 = stump_fit(xs, residual)
def boosted(x):
return stump_predict(s1, x) + lr * stump_predict(s2, x)
def mse(preds):
return sum((y - p) ** 2 for y, p in zip(ys, preds)) / len(ys)
print('MSE stump1:', round(mse(pred1), 3))
print('MSE boosted:', round(mse([boosted(x) for x in xs]), 3))
The first stump alone leaves a mean squared error of one. Fit a second stump to the residuals it left behind, add that correction at half weight, and the error drops to roughly half. No individual tree got more powerful; the gain came entirely from the second tree mopping up what the first missed, and a third or fourth round would keep eating into the remainder. That steady, additive shrinkage is the signature of boosting. It is also why boosting has a learning rate: adding each tree at full strength would let the ensemble chase individual points and overfit, so each correction is scaled down and the model learns cautiously, one small fix at a time.
flowchart LR
subgraph BG["bagging"]
BA["many independent trees"] --> BV["lowers variance"]
end
subgraph BS["boosting"]
BB["sequential corrective trees"] --> BI["lowers bias"]
end
style BV fill:#166534,color:#fff
style BI fill:#b45309,color:#fff
When to reach for which
Both methods combine trees, but they fail in opposite ways, and that decides when you use them. Bagging is robust and almost immune to overfitting: add more trees and the error simply plateaus, it never grows worse, and its members train in parallel so it scales easily. Reach for it when the data is noisy and a single tree is unstable. Boosting is hungry and precise: it squeezes out bias that bagging cannot touch, and on clean, structured data it often reaches the highest accuracy. But it pays for that precision with fragility. Add too many trees, or set the learning rate too high, and boosting will happily memorise the training set, so it must be tuned carefully where a random forest is forgiving.
What does this print? The function returns the prediction that appears most often.
def majority_vote(predictions):
counts = {}
for p in predictions:
counts[p] = counts.get(p, 0) + 1
return max(counts, key=counts.get)
print(majority_vote([0, 1, 1, 0, 1]))
Count how many times each value appears: 0 appears twice, 1 appears three times.
The vote goes to the value with the higher count.
Write bootstrap_sample(items, rng) that returns a new list the same length as items, where each entry is drawn with replacement using rng.randrange(len(items)). This resampling with replacement is the bootstrap step bagging is built on.
def bootstrap_sample(items, rng):
# draw len(items) items, each picked at random with replacement
pass
Use a list comprehension that runs len(items) times.
Each draw picks items[rng.randrange(len(items))]. The same index can be picked more than once.
def bootstrap_sample(items, rng):
return [items[rng.randrange(len(items))] for _ in range(len(items))]
Boosting fits each new tree to the residual — the error left over after the current model's prediction. This residuals(targets, predictions) has the subtraction backwards, so the next tree would learn to repeat the error instead of fixing it. Fix it so each residual is target - prediction.
def residuals(targets, predictions):
# the leftover error: what the model still needs to learn
return [p - t for t, p in zip(targets, predictions)]
The residual is how much the prediction undershoots or overshoots the target.
Swap the order so each term is target minus prediction.
def residuals(targets, predictions):
return [t - p for t, p in zip(targets, predictions)]
Write bag_average(model_preds) that turns a list of per-model prediction lists into the bagged prediction: the average across models at each position. model_preds is a list of equal-length lists, one per model. This aggregation step is what reduces variance.
def bag_average(model_preds):
# average across models at each position
pass
zip(*model_preds) transposes the lists so each group holds one position across all models.
Sum each group and divide by the number of models.
def bag_average(model_preds):
n = len(model_preds)
return [sum(col) / n for col in zip(*model_preds)]
Your single decision tree has high variance — small changes to the training data produce very different trees. Which ensemble method most directly attacks that variance?
Bagging trains many trees on different bootstrap samples and averages them; independent errors cancel, which directly lowers variance. Boosting instead lowers bias by sequentially fixing residual errors.
Recap
- A single tree has high variance — small changes to the data move it a lot.
- Bagging trains many trees on different bootstrap samples and averages them; independent errors cancel, lowering variance. A random forest adds random feature subsets so the trees disagree more.
- Boosting trains trees in sequence, each fitting the residual errors of the running total; it lowers bias by stacking weak learners.
- Reach for bagging when models are unstable or data is noisy; reach for boosting when you need to squeeze out bias on cleaner data.
- Bagging is forgiving and hard to overfit; boosting is powerful but can overfit, so tune its learning rate and tree count with care.