You ran cross-validation, swept five values of a hyperparameter, kept the best-scoring one, and reported that same best score as the model's expected accuracy. The number looks rigorous — it came from folds — yet it is quietly optimistic. This is the model-selection twin of the leakage you met when preprocessing touched the test set: the data used to CHOOSE the model is also the data used to GRADE it. Cross-validation shields you from one lucky split, but it cannot shield you from yourself. The instant you scan a grid of candidates and keep the winner, that winning score absorbs a share of good fortune that will not survive genuinely new data. Tuning is necessary; the trap is presenting the tuning score as though it were an honest verdict.
flowchart LR A["grid of lambdas"] --> B["CV-score each<br/>on the same folds"] B --> C["keep the min<br/>(best scorer)"] C --> D["report that score<br/>as expected performance"] D --> E["score is optimistic:<br/>winner got lucky"] style C fill:#b45309,color:#fff style E fill:#b91c1c,color:#fff
Name the parts
Three terms keep the discussion precise. A hyperparameter is a setting you fix BEFORE training — the neighbour count in k-NN, the penalty strength in ridge regression, the depth of a tree. The model never learns it from data; you supply it. A grid is the list of candidate values you intend to try. Cross-validation splits the training data into k folds, trains on k-1 of them, scores on the held fold, and repeats so every example is tested once; the mean across folds is the CV score for that candidate. The decision under the microscope is what happens when the very fold system that produces these scores is also asked to declare the winner.
Why the winner is too good
Each fold score is a noisy estimate of true performance — slightly high or low depending on which examples happened to land in the held fold. When you take the best score across a grid, you scoop up the lucky-good lows along with any genuine signal. Statisticians call this the winner's curse: the selected value looks better than it truly is, because part of its victory was random noise pointing favourably. Trying ten candidates instead of three deepens the curse, not cures it, because you hand noise more chances to manufacture a flattering low. A single CV score, reported honestly, is a fine estimate; the bias enters the moment you take the minimum over several candidates and treat that minimum as one honest measurement.
# Each candidate's true score is 10.0; the CV estimate is true score + fold noise.
estimates = [10.0 + noise for noise in (0.3, -0.8, 0.1, -0.4, 0.6)]
best = min(estimates)
print("true score of every candidate:", 10.0)
print("best CV estimate reported:", best)
print("optimism gap:", round(10.0 - best, 2))
Run the grid and a table of CV scores appears, one row per candidate. The lowest number is the best mean fold error this grid can buy — call its lambda the winner. Nothing about that winner is wrong; it is a defensible choice. The error is one step later, when that lowest number is copied into a report as the model's expected error. The table was built by asking the folds to rank candidates, so the winning row carries the optimism of having won a contest judged against itself. Keep the winner for your model; distrust its score for your estimate.
import random
random.seed(1)
xs = list(range(1, 21))
ys = [3 * x + random.uniform(-4, 4) for x in xs]
def folds(n, k):
idx = list(range(n))
return [idx[i::k] for i in range(k)]
def ridge_w(xt, yt, lam):
return sum(x * y for x, y in zip(xt, yt)) / (sum(x * x for x in xt) + lam)
def cv_mse(xs, ys, lam, k=5):
fs = folds(len(xs), k)
err, n = 0.0, 0
for fi in range(k):
te = fs[fi]
tr = [i for j in range(k) if j != fi for i in fs[j]]
w = ridge_w([xs[i] for i in tr], [ys[i] for i in tr], lam)
for i in te:
err += (w * xs[i] - ys[i]) ** 2
n += 1
return err / n
for lam in (0, 2, 10, 50, 200):
print("lam", lam, "cv_mse", round(cv_mse(xs, ys, lam), 2))
The cleanest way to feel the optimism is to hold one more set of data in reserve — a test set the grid never touched. Tune your lambda on the folds, note its CV score, then evaluate that chosen lambda on the reserved set. The reserved score is almost always worse than the CV score that elected it, and the gap widens as the grid grows. That gap IS the leakage, made visible: the folds answered two questions at once — which lambda is best, and how good is the best lambda — and answering both with the same numbers is what bends the result in the model's favour.
flowchart TD D["all data"] --> O["outer folds<br/>(for scoring)"] O --> O1["outer train<br/>of fold i"] O1 --> I["inner folds<br/>(for tuning)"] I --> T["pick best lambda"] T --> R["refit on outer train"] R --> S["score on outer test fold"] S --> M["mean of outer scores =<br/>honest estimate"] style I fill:#3776ab,color:#fff style M fill:#166534,color:#fff
Nested cross-validation
The remedy lets two different slices of data answer those two questions. Split the data into k outer folds. For each outer fold in turn, hold it out as a test set and work only with the remaining outer-training data. Inside that outer-training data, run a second, inner cross-validation across your grid to choose the best lambda — the inner folds never see the outer test fold. Refit one model on all the outer-training data with the chosen lambda, and score it exactly once on the held-out outer test fold. Repeat across every outer fold and average those outer scores. Because each outer test fold stayed sealed while its lambda was chosen, the average is an honest estimate of how a tuning procedure — not a single lucky lambda — performs on new data.
import random
random.seed(7)
xs = list(range(1, 21))
ys = [3 * x + random.uniform(-5, 5) for x in xs]
def folds(n, k):
idx = list(range(n))
return [idx[i::k] for i in range(k)]
def ridge_w(xt, yt, lam):
return sum(x * y for x, y in zip(xt, yt)) / (sum(x * x for x in xt) + lam)
def inner_cv(train_idx, lam, k=5):
fs = folds(len(train_idx), k)
err, n = 0.0, 0
for fi in range(k):
te = fs[fi]
tr = [i for j in range(k) if j != fi for i in fs[j]]
w = ridge_w([xs[train_idx[i]] for i in tr], [ys[train_idx[i]] for i in tr], lam)
for i in te:
err += (w * xs[train_idx[i]] - ys[train_idx[i]]) ** 2
n += 1
return err / n
def nested(grid, k=5):
fs = folds(len(xs), k)
err, n = 0.0, 0
for fi in range(k):
outer_te = fs[fi]
outer_tr = [i for j in range(k) if j != fi for i in fs[j]]
best_lam, best = None, None
for lam in grid:
sc = inner_cv(outer_tr, lam, k)
if best is None or sc < best:
best, best_lam = sc, lam
w = ridge_w([xs[i] for i in outer_tr], [ys[i] for i in outer_tr], best_lam)
for i in outer_te:
err += (w * xs[i] - ys[i]) ** 2
n += 1
return err / n
print("nested estimate:", round(nested([0, 2, 10, 50, 200]), 2))
Notice what the nested estimate is, and is not. It estimates the whole pipeline — grid, inner CV, refit — applied to a fresh dataset of this size. It is NOT the score of the single lambda you will eventually ship. Once nesting has told you the procedure is sound, you typically run one final inner CV on ALL the data to pick the production lambda. The discipline that matters is that the number you report to stakeholders comes from the outer loop, which never let the test folds vote on their own lambda. A non-nested CV score lets the test folds vote, which is precisely the optimism this lesson exists to remove.
flowchart LR A["non-nested:<br/>tune and score on same folds"] --> B["optimistic score"] C["nested:<br/>tune on inner, score on outer"] --> D["honest score"] style B fill:#b91c1c,color:#fff style D fill:#166534,color:#fff
What does this print? A strided 5-fold split of 10 indices.
def folds(n, k):
idx = list(range(n))
return [idx[i::k] for i in range(k)]
fs = folds(10, 5)
print(len(fs), [len(f) for f in fs])
folds(10, 5) takes every 5th index starting at 0, then 1, then 2, and so on.
Ten indices split into five strided folds give five folds of two.
Write kfold(n, k, fold_index) that returns (train_idx, test_idx) for the fold_index-th fold (0-based), using a strided split where the test fold is list(range(n))[fold_index::k].
def kfold(n, k, fold_index):
# test fold = indices[fold_index::k]; train = the rest
pass
Slice the index list with [fold_index::k] to get the test fold.
Build train from every index not in that test fold.
def kfold(n, k, fold_index):
idx = list(range(n))
test = idx[fold_index::k]
test_set = set(test)
train = [i for i in idx if i not in test_set]
return train, test
This function picks the best lambda by scoring candidates on the TEST fold, then reports that same test fold's score — tuning and grading on the same data. Fix it so lambda is chosen on the TRAINING fold only, and the test fold is used only for the final score.
def fit(xt, yt, lam):
return sum(x*y for x,y in zip(xt,yt)) / (sum(x*x for x in xt) + lam)
def mse(pred, true):
return sum((p-t)**2 for p,t in zip(pred,true)) / len(true)
def leaked_score(xs, ys, train_idx, test_idx, grid):
# BUG: tunes lambda on the TEST fold and scores the winner on that same test fold
te_x = [xs[i] for i in test_idx]; te_y = [ys[i] for i in test_idx]
best = min(grid, key=lambda lam: mse([fit(te_x, te_y, lam)*x for x in te_x], te_y))
w = fit(te_x, te_y, best)
return round(mse([w*x for x in te_x], te_y), 3)
The min(grid, key=...) must score candidates on the TRAINING fold, not the test fold.
Refit the weight on the training fold too; only the final mse uses the test fold.
def fit(xt, yt, lam):
return sum(x*y for x,y in zip(xt,yt)) / (sum(x*x for x in xt) + lam)
def mse(pred, true):
return sum((p-t)**2 for p,t in zip(pred,true)) / len(true)
def leaked_score(xs, ys, train_idx, test_idx, grid):
tr_x = [xs[i] for i in train_idx]; tr_y = [ys[i] for i in train_idx]
best = min(grid, key=lambda lam: mse([fit(tr_x, tr_y, lam)*x for x in tr_x], tr_y))
w = fit(tr_x, tr_y, best)
te_x = [xs[i] for i in test_idx]; te_y = [ys[i] for i in test_idx]
return round(mse([w*x for x in te_x], te_y), 3)
Write nested_cv_score(xs, ys, grid, k) performing k-fold OUTER cross-validation. For each outer fold: hold out the outer test fold, choose the best lambda from grid by inner k-fold CV on the outer-TRAINING indices only, refit on all outer-training data with that lambda, and score once on the outer test fold. Return the mean squared error averaged over every outer test point. The helpers _folds, _ridge_w, and _inner_cv are provided.
def _folds(n, k):
idx = list(range(n))
return [idx[i::k] for i in range(k)]
def _ridge_w(xt, yt, lam):
return sum(x*y for x,y in zip(xt,yt)) / (sum(x*x for x in xt) + lam)
def _inner_cv(xs, ys, train_idx, lam, k):
fs = _folds(len(train_idx), k)
err, n = 0.0, 0
for fi in range(k):
te = fs[fi]
tr = [i for j in range(k) if j != fi for i in fs[j]]
w = _ridge_w([xs[train_idx[i]] for i in tr], [ys[train_idx[i]] for i in tr], lam)
for i in te:
err += (w*xs[train_idx[i]] - ys[train_idx[i]])**2; n += 1
return err / n
def nested_cv_score(xs, ys, grid, k=5):
# For each OUTER fold: hold it out, pick the best lambda from `grid`
# by inner CV on the outer-training indices only, refit on all outer
# training data, and score on the outer test fold.
pass
Use _folds(len(xs), k) for the outer folds; the test fold is fs[fi].
Pick best_lam by calling _inner_cv on the outer-training indices, then refit and score on the outer test fold.
def _folds(n, k):
idx = list(range(n))
return [idx[i::k] for i in range(k)]
def _ridge_w(xt, yt, lam):
return sum(x*y for x,y in zip(xt,yt)) / (sum(x*x for x in xt) + lam)
def _inner_cv(xs, ys, train_idx, lam, k):
fs = _folds(len(train_idx), k)
err, n = 0.0, 0
for fi in range(k):
te = fs[fi]
tr = [i for j in range(k) if j != fi for i in fs[j]]
w = _ridge_w([xs[train_idx[i]] for i in tr], [ys[train_idx[i]] for i in tr], lam)
for i in te:
err += (w*xs[train_idx[i]] - ys[train_idx[i]])**2; n += 1
return err / n
def nested_cv_score(xs, ys, grid, k=5):
fs = _folds(len(xs), k)
err, n = 0.0, 0
for fi in range(k):
outer_te = fs[fi]
outer_tr = [i for j in range(k) if j != fi for i in fs[j]]
best_lam, best = None, None
for lam in grid:
sc = _inner_cv(xs, ys, outer_tr, lam, k)
if best is None or sc < best:
best, best_lam = sc, lam
w = _ridge_w([xs[i] for i in outer_tr], [ys[i] for i in outer_tr], best_lam)
for i in outer_te:
err += (w*xs[i] - ys[i])**2; n += 1
return err / n
Which procedure gives an honest estimate of how a tuned model will perform on new data?
Only nested CV keeps the data that chooses the hyperparameter (inner folds) separate from the data that scores it (outer folds). The other options either grade on the same folds that tuned, or evaluate on data the model trained on.
Recap
- Tuning on the scoring folds leaks. Selecting a hyperparameter by the same CV score you report bends the estimate upward — the winner's curse.
- More candidates mean more bias. Taking the best of a grid absorbs lucky-good folds; a wide grid is a wider chance for noise to flatter you.
- The optimism is measurable. Evaluate the chosen value on data the grid never saw, and watch its score fall below the one that elected it.
- Nested CV fixes it. An outer loop scores; an inner loop tunes. Each outer test fold is sealed while its own lambda is chosen, so the outer mean is honest.
- The number you report should come from data that never voted on the model. Next you will see why a random split can itself leak — on grouped and time-ordered data.