When you trained models by hand, you made quiet choices that changed the result: how many steps gradient descent ran, how deep the decision tree grew, how many neighbours k-NN consulted. Those choices are not learned from data the way weights are; you set them up front, and they are called hyperparameters to distinguish them from the parameters the model fits during training.
Pick them badly and a capable algorithm underperforms; pick them well and an ordinary one shines. This lesson is about searching that space honestly — trying combinations, scoring each, and reading the winner without fooling yourself.
flowchart LR A["n_neighbors: 1, 3, 5"] --> C["grid of candidates"] B["weights: uniform, distance"] --> C C --> D["3 x 2 = 6 fits"] style C fill:#3776ab,color:#fff
Grid search: try them all
GridSearchCV takes an estimator and a dictionary of the values you want to try for each hyperparameter, then fits a model for every combination and scores it with cross-validation. The grid is the cartesian product of the lists you give it: three choices for one knob and two for another make six candidates, each evaluated on its own merits.
Because every candidate is judged by the same cross-validation, the comparison is apples-to-apples. The search returns best_params_, the combination that won, and best_score_, the mean CV score it won with. The heavy lifting — looping, folding, refitting — is the library's; the judgment about what to search and how to read it is yours.
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
X, y = load_iris(return_X_y=True)
params = {'n_neighbors': [1, 3, 5]}
gs = GridSearchCV(KNeighborsClassifier(), params, cv=5)
gs.fit(X, y)
print(gs.best_params_)
print(round(gs.best_score_, 3))
What cv_results_ reveals
Beyond the single winner, GridSearchCV stores the full table of every candidate's score in cv_results_. That table is more useful than the headline, because it shows the shape of the search: which hyperparameters mattered, which were flat, and whether the winner sits at the edge of the grid — a hint that you should widen the search in that direction.
A winner pinned to the boundary of your grid often means the true optimum lies outside it, and the honest move is to expand the range and search again rather than trust a number that only won because you stopped looking.
flowchart TD G["grid of candidates"] --> CV["each scored by k-fold CV"] CV --> B["best_params_, best_score_"] B -.->|"validation estimate, not a test score"| H["hold out a separate test set"] style H fill:#166534,color:#fff
Reading the results without fooling yourself
The number that matters most is what best_score_ means. It is the mean cross-validation score of the winning parameter combination, a validation estimate reached by averaging several held-out folds. It is NOT a score on your final test set, and because you used it to choose the winner, it carries a small optimistic bias: among the many candidates you tried, some looked good partly by chance.
That is why a disciplined workflow holds out one more split, a test set the search never saw, and reports that score as the honest headline. Trust the search to pick parameters; trust the untouched test set to tell you whether they truly generalise.
# A by-hand hyperparameter sweep: try each k, keep the best on validation.
params = [1, 3, 5, 7]
val_scores = [0.78, 0.86, 0.91, 0.88]
best_p, best_s = params[0], val_scores[0]
for p, s in zip(params, val_scores):
if s > best_s:
best_p, best_s = p, s
print('best param:', best_p, 'score:', best_s)
How big should a grid be
Every candidate in a grid pays for a full cross-validation, so the cost grows as the product of the lists. Three values for one knob, four for another, and five folds means sixty fits, and that is before you add a second model. A grid is a budget, not a wish list: start small and seeded, find which knobs move the score at all, and only then expand the promising ones.
For large spaces the library offers RandomizedSearchCV, which samples combinations instead of exhausting them, trading a guarantee of checking every cell for the ability to explore many more of them. Exhaustive grids suit small, well-understood spaces; sampling suits combinations that run into the thousands.
The trap: tuning on the test set
The temptation, once you have a test set, is to loop over hyperparameters and keep whichever scores highest on it. That is leakage wearing a lab coat. Each time you peek at the test score and adjust a choice because of it, the test set quietly becomes part of your tuning, and its score drifts upward for the same reason the Module 2 leak did — the data is no longer unseen.
Run the block below and watch the divergence: the parameter that looks best on the test set is not the one that looks best on validation, because the test set is a single noisy draw and you have cherry-picked its favourable fluctuations.
# Picking the winner by TEST score leaks the test set into the choice.
params = [1, 5, 9]
val_scores = [0.70, 0.90, 0.85]
test_scores = [0.95, 0.80, 0.78]
best_by_val = params[val_scores.index(max(val_scores))]
best_by_test = params[test_scores.index(max(test_scores))]
print('by validation:', best_by_val)
print('by test:', best_by_test)
flowchart LR D["all data"] --> T["train: learn"] D --> V["validation: tune"] D --> S["test: report once"] style S fill:#b45309,color:#fff
# Three-way split by hand: train to learn, validation to tune, test to report. import random random.seed(42) rows = list(range(10)) random.shuffle(rows) n = len(rows) train = rows[:int(n * 0.6)] valid = rows[int(n * 0.6):int(n * 0.8)] test = rows[int(n * 0.8):] print(len(train), len(valid), len(test))
After gs.fit(X, y), what does gs.best_score_ represent?
best_score_ is the mean CV score of the winning combination during the search. It is a validation estimate used to pick the winner, not a test-set score, which is why reporting it as the final result is optimistic.
How many combinations does this grid contain? A grid is the cartesian product of the value lists. (Build-verified; the browser cannot run scikit-learn.)
from sklearn.model_selection import ParameterGrid
grid = {'n_neighbors': [1, 3, 5], 'weights': ['uniform', 'distance']}
print(len(list(ParameterGrid(grid))))
Count the values in each list: 3 choices for n_neighbors, 2 for weights.
A cartesian product multiplies them: 3 times 2.
Write select_param(params, val_scores) that returns the param with the highest validation score. params and val_scores are equal-length lists. On a tie, keep the earliest (first) one — exactly what a stable sweep does.
def select_param(params, val_scores):
# return params[i] where val_scores[i] is highest (first wins ties)
pass
Track the best param and its score as you loop, using strict greater-than so ties keep the first.
Start from params[0], val_scores[0] and update only when a strictly higher score appears.
def select_param(params, val_scores):
best_p, best_s = params[0], val_scores[0]
for p, s in zip(params, val_scores):
if s > best_s:
best_p, best_s = p, s
return best_p
This chooser picks the model with the best TEST score, which leaks the test set into the decision. Fix it to pick by the VALIDATION score instead — the test set must stay untouched until the final report.
def pick_model(params, val_scores, test_scores):
# BUG: winner chosen by best TEST score -> leaks the test set
return params[test_scores.index(max(test_scores))]
The winner should come from val_scores, not test_scores.
Use val_scores.index(max(val_scores)) to find the position, then index params.
def pick_model(params, val_scores, test_scores):
return params[val_scores.index(max(val_scores))]
After picking hyperparameters with GridSearchCV, why score ONE more time on a held-out test set the search never saw?
best_score_ is biased upward because the search picked the winner after seeing the CV scores. A test set the search never touched was not used to choose anything, so its score is the unbiased estimate you should report.
Recap
- Hyperparameters are the choices you set before training (tree depth, k, regularisation strength); parameters are what the model learns.
- GridSearchCV fits every combination in the grid and scores each by cross-validation, returning
best_params_andbest_score_. best_score_is an optimistic CV estimate, not a final test score, because you picked the winner after seeing the scores.- Hold out a separate test set the search never touched, and report its score once — tuning on the test set is leakage in a lab coat.
That closes the loop from data to an honest, leakage-free model. The remaining lessons step back to decide, for a real problem, whether classical ML, fine-tuning, or prompting is even the right tool — and then to ship a capstone.