Module 8 · Overfitting & Honest Validation ⏱ 19 min

K-Fold Cross-Validation

By the end of this lesson you will be able to:
  • Explain why a single train/test split gives a high-variance estimate of performance
  • Implement k-fold cross-validation that rotates every example into a test fold exactly once
  • Report a mean score and a spread across folds to express uncertainty
  • Recognise that cross-validation is for model selection, with a final test set still held back

You split your data into train and test, trained a model, and got a score of 0.87. The number feels solid until you change the random seed and the same model on the same data now scores 0.81. Which one is real? Neither, on its own. A single split is one draw from a distribution of possible splits, and the score it produces swings as much as the split does. Reporting that swing as if it were a fact is how teams ship a model that looks brilliant on a Tuesday and mediocre on a Wednesday. Cross-validation is the fix: instead of betting everything on one cut of the data, cut it several different ways, score each, and report the average and the spread. The average is your estimate; the spread is your honesty about how sure you can be.

flowchart TD
  D["dataset"] --> S["split into k folds"]
  S --> F1["fold 1: test on 1,<br/>train on the rest"]
  S --> F2["fold 2: test on 2,<br/>train on the rest"]
  S --> F3["fold k: test on k,<br/>train on the rest"]
  F1 --> A["average the k scores"]
  F2 --> A
  F3 --> A
  style A fill:#10b981,color:#fff
K-fold splits the data into k parts; each part takes a turn as the test set while the rest trains.

How k-fold works

The recipe is simple and every example earns its keep. Split the dataset into k roughly equal parts called folds. For each fold in turn, hold that fold out as the test set and train on the remaining k - 1 folds; record the score. After k rounds, every example has been in the test set exactly once and in the training set k - 1 times — nothing is wasted, nothing is double-counted. The final estimate is the mean of the k scores. Five and ten folds are the usual choices; a larger k means more training data per fold (less biased estimates) but more rounds and more correlation between folds. Five folds is a fine default that balances the two, and it is what most practitioners reach for first.

Build the k folds as train/test index lists. Every index lands in a test fold exactly once. Press Run.
def k_folds(n, k):
    fold_size = n // k
    splits = []
    for fold in range(k):
        start = fold * fold_size
        end = start + fold_size if fold < k - 1 else n
        test = list(range(start, end))
        train = list(range(0, start)) + list(range(end, n))
        splits.append((train, test))
    return splits

for i, (train, test) in enumerate(k_folds(10, 5)):
    print("fold", i, "test", test, "train", train)

Why every example earns its keep

Notice the structure of those folds. Each test fold is a disjoint slice, and together the five test folds cover all ten indices with no overlaps and no gaps. That is the efficiency that makes cross-validation worth its cost: you evaluate on the whole dataset without ever testing on data the model trained on. The last fold soaks up any remainder when n does not divide evenly by k, which keeps the folds as even as possible. In practice you shuffle the data before cutting it, so that sorted or grouped records do not all land in the same fold — a fixed seed keeps the whole procedure reproducible, exactly as with a single split.

flowchart LR
  O["one split"] --> V1["one number, high variance<br/>depends on the lucky cut"]
  K["k-fold"] --> V2["k numbers, averaged<br/>lower variance, plus a spread"]
  style O fill:#b45309,color:#fff
  style K fill:#166534,color:#fff
One split gives one wobbly number; k-fold gives an average and a spread.
Full 5-fold CV with a linear model: per-fold MSE, the mean, and the spread. Run it.
import random

def k_folds_shuffled(n, k, seed):
    indices = list(range(n))
    random.seed(seed)
    random.shuffle(indices)
    fold_size = n // k
    splits = []
    for fold in range(k):
        start = fold * fold_size
        end = start + fold_size if fold < k - 1 else n
        test = indices[start:end]
        train = indices[:start] + indices[end:]
        splits.append((train, test))
    return splits

def fit_line(xs, ys):
    n = len(xs)
    mx = sum(xs) / n
    my = sum(ys) / n
    slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
    intercept = my - slope * mx
    return slope, intercept

def mse(slope, intercept, xs, ys):
    return sum((intercept + slope * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)

xs = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
ys = [2.1, 3.9, 6.2, 7.8, 10.3, 11.7, 14.2, 15.9, 18.1, 19.8]

scores = []
for train_i, test_i in k_folds_shuffled(10, 5, 42):
    tr_x = [xs[i] for i in train_i]
    tr_y = [ys[i] for i in train_i]
    te_x = [xs[i] for i in test_i]
    te_y = [ys[i] for i in test_i]
    s, b = fit_line(tr_x, tr_y)
    scores.append(round(mse(s, b, te_x, te_y), 3))

print("fold MSEs:", scores)
print("mean:", round(sum(scores) / len(scores), 3))
print("spread: max - min =", round(max(scores) - min(scores), 3))

The mean and the spread

Run the full cross-validation and you get five numbers, one per fold. Their mean is your performance estimate; their spread (the distance from the worst fold to the best) is your warning label. A small spread means the model performs consistently regardless of which examples it was tested on — you can trust the mean. A large spread means the model's quality depends heavily on the draw, and the mean is hiding a wide range of possible realities. That spread is information a single split could never give you, because a single split has no other folds to compare against. When you report a model's score to anyone, the mean plus the spread is the honest version of the story.

One split wobbles a lot across seeds; the k-fold mean stays steady. Run it.
import random

def k_folds_shuffled(n, k, seed):
    indices = list(range(n))
    random.seed(seed)
    random.shuffle(indices)
    fold_size = n // k
    splits = []
    for fold in range(k):
        start = fold * fold_size
        end = start + fold_size if fold < k - 1 else n
        test = indices[start:end]
        train = indices[:start] + indices[end:]
        splits.append((train, test))
    return splits

def fit_line(xs, ys):
    n = len(xs)
    mx = sum(xs) / n
    my = sum(ys) / n
    slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
    return slope, my - slope * mx

def mse(slope, intercept, xs, ys):
    return sum((intercept + slope * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)

xs = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
ys = [2.1, 3.9, 6.2, 7.8, 10.3, 11.7, 14.2, 15.9, 18.1, 19.8]

def single(seed):
    idx = list(range(10))
    random.seed(seed)
    random.shuffle(idx)
    tr, te = idx[:8], idx[8:]
    s, b = fit_line([xs[i] for i in tr], [ys[i] for i in tr])
    return round(mse(s, b, [xs[i] for i in te], [ys[i] for i in te]), 3)

def cv_mean(seed):
    scores = []
    for tr, te in k_folds_shuffled(10, 5, seed):
        s, b = fit_line([xs[i] for i in tr], [ys[i] for i in tr])
        scores.append(mse(s, b, [xs[i] for i in te], [ys[i] for i in te]))
    return round(sum(scores) / len(scores), 3)

seeds = [1, 2, 3, 4, 5]
single_scores = [single(s) for s in seeds]
cv_scores = [cv_mean(s) for s in seeds]
print("single splits:", single_scores, " spread", round(max(single_scores) - min(single_scores), 3))
print("cv means:    ", cv_scores, " spread", round(max(cv_scores) - min(cv_scores), 3))
flowchart LR
  D["all data"] --> H["held-out test set<br/>locked away until the end"]
  D --> C["cross-validation on the rest<br/>choose model and hyperparameters"]
  C --> F["final: evaluate once<br/>on the held-out test set"]
  H --> F
  style H fill:#166534,color:#fff
  style C fill:#3776ab,color:#fff
Cross-validation chooses the model, but a held-out test set judges it once at the end.
Exercise

What does this print? Fold 2's test indices for 10 examples split into 5 folds of 2.

def test_fold(n, k, fold):
    fold_size = n // k
    start = fold * fold_size
    end = start + fold_size if fold < k - 1 else n
    return list(range(start, end))

print(test_fold(10, 5, 2))
Exercise

Write test_fold(n, k, fold) that returns the test indices for a single fold of a k-fold split of range(n). Use fold_size = n // k; the last fold (index k - 1) extends to n to absorb any remainder.

def test_fold(n, k, fold):
    # the test indices [start, end) for this fold
    pass
Exercise

What does this print? The cross-validation estimate is the mean of the per-fold scores.

scores = [3.0, 5.0, 4.0, 2.0, 6.0]
print(round(sum(scores) / len(scores), 2))
Exercise

This function is given a list of per-fold scores and should return the cross-validation MEAN, but it only returns the first fold's score — ignoring the other four. That is exactly the one-split-can-mislead trap in disguise. Fix it to average all the folds.

def cv_mean_score(scores):
    return scores[0]
Exercise

Write train_indices_for_fold(test_folds, fold) that returns the TRAIN indices for a given fold. test_folds is a list of test-index lists, one per fold (they partition the index set). The train set is every index that is NOT in test_folds[fold], returned in sorted order.

def train_indices_for_fold(test_folds, fold):
    # all indices except those in test_folds[fold]
    pass

Recap

  • A single train/test split is one draw from many possible splits; its score swings with the seed, so reporting it alone is a gamble.
  • K-fold cross-validation cuts the data into k folds, tests on each in turn, and reports the mean of the k scores.
  • Every example is in a test fold exactly once and a training fold k - 1 times — the whole dataset is used for both training and evaluation, with no leakage.
  • The spread across folds is your honesty: a small spread means a trustworthy mean, a large spread means the model's quality depends on the draw.
  • Cross-validation is a selection tool for choosing models and hyperparameters; it must not be reported as final performance, which needs a separate untouched test set evaluated once.
  • The next lesson tackles what happens when selection itself leaks into evaluation, and how nested cross-validation closes the gap.

Checkpoint quiz

Why does a single train/test split give an unreliable performance estimate?

After running 5-fold cross-validation, how should you report the model's estimated performance?

You use 5-fold cross-validation to pick the best value of lambda, then report that best cross-validated mean as the model's final accuracy. What is wrong with this?

Go deeper — technical resources