In Module 2 you met your first leakage trap: scaling the whole dataset before splitting it lets information from the test rows leak into the training step. The model is then graded on data it has already seen a shadow of, the holdout score shoots up, and the whole evaluation becomes a comforting lie that collapses the moment the model meets real users.
The trap is easy to understand and notoriously hard to avoid by willpower alone, because the leaky code looks perfectly natural — fit the scaler, transform, then split. This lesson shows the structural fix: a scikit-learn Pipeline that makes the mistake impossible to write.
flowchart TD A["fit scaler on ALL data"] --> B["test statistics leak into training"] B --> C["holdout score is inflated"] C --> D["shock when the model hits production"] style B fill:#b91c1c,color:#fff style C fill:#b91c1c,color:#fff
The leaky way, and why it feels innocent
Here is the bug in its most common form. You create a StandardScaler, call .fit_transform() on the entire feature matrix, and only afterwards split into train and test. Every row, including the ones destined for the holdout, contributed to the mean and standard deviation the scaler learned. When the model is later scored on the test rows, those rows have already shaped the very transformation applied to them.
The score you read is no longer an honest estimate of unseen data; it is the model recognising its own handwriting. Worse, nothing in the code raises an error, so the bug is silent, which is exactly why it survives in production notebooks for years.
from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # LEAK: fit on ALL rows, test included model = LogisticRegression(max_iter=1000) model.fit(X_scaled, y) # trained on leaked scaling
The Pipeline: leakage-proof by construction
A Pipeline chains preprocessing and the model into a single object that exposes the same fit / predict / score verbs you met in the last lesson. The difference is what happens inside cross-validation: for each fold, scikit-learn fits the scaler on only that fold's training rows, transforms them, and trains the model on the result, then evaluates on the untouched validation rows.
Because the scaling is sealed inside the fold, the validation rows can never influence the statistics used to scale them. The Module 2 bug is not avoided by remembering to be careful; it is made structurally impossible. You cannot accidentally fit the scaler on the test set, because the Pipeline never hands you the chance.
from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.model_selection import cross_val_score pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) scores = cross_val_score(pipe, X, y, cv=5) print(round(scores.mean(), 3))
flowchart LR A["raw fold"] --> B["StandardScaler"] B --> C["LogisticRegression"] C --> D["score"] B -.->|"fit only on this fold train rows"| E["no leakage"] style E fill:#166534,color:#fff
# Standard scaling (z-scores), built by hand (Module 2).
def standardize(xs):
mean = sum(xs) / len(xs)
var = sum((x - mean) ** 2 for x in xs) / len(xs)
std = var ** 0.5
return [(x - mean) / std for x in xs]
print([round(v, 3) for v in standardize([2.0, 4.0, 6.0, 8.0])])
What StandardScaler actually does
Run the block and you will see the by-hand version of every StandardScaler. For each column it subtracts the mean and divides by the standard deviation, leaving a feature with mean zero and spread one. Distance- and gradient-based models are sensitive to scale, because a feature measured in thousands will otherwise dominate one measured in decimals, no matter which carries more signal.
The z-scores you see are exactly what the library computes. The only question this lesson adds is where that mean and standard deviation come from, and the answer must be the training rows alone, or the columns quietly smuggle test information into training.
Why this is airtight
The reason a Pipeline works is that cross_val_score does not see a scaler and a model separately; it sees one estimator. To score a fold, it calls that estimator's fit on the training portion and score on the validation portion, and the estimator happens to be a chain that scales-then-fits internally. There is no step at which the validation rows are allowed near the scaler's fit.
That is the whole game. Leakage is a problem of ordering: fitting global statistics before splitting. A Pipeline fixes the ordering by binding the steps together, so the only way to leak is to break the Pipeline apart, which is a conspicuous act rather than a silent default.
# The leakage, made visible: the centering mean shifts when test rows join. train = [10, 20, 30, 40] test = [1000, 2000] mean_all = sum(train + test) / len(train + test) mean_train = sum(train) / len(train) print(round(mean_all, 2)) print(round(mean_train, 2))
flowchart TD F["fold 1: scaler + model fit on train-1"] G["fold 2: scaler + model fit on train-2"] H["fold 3: scaler + model fit on train-3"] F --> M["mean of fold scores"] G --> M H --> M style M fill:#3776ab,color:#fff
Cross-validation, briefly
A single train/test split can flatter or punish you by the luck of the draw, so honest evaluation repeats the split. K-fold cross-validation chops the data into k pieces, trains on k minus one of them, validates on the held-out one, and rotates until every piece has been the validation set exactly once. The reported score is the mean across folds.
When the thing being scored is a Pipeline, every fold builds its own fresh scaler-model pair on that fold's training rows. That is why the figure shows three separate fits rather than one: leakage is avoided per fold, not once globally.
# A by-hand train/test split with a fixed seed (Module 2). import random random.seed(42) rows = list(range(10)) random.shuffle(rows) n = len(rows) train = rows[:int(n * 0.6)] test = rows[int(n * 0.6):] print(len(train), len(test))
Why does calling scaler.fit_transform(X) on the full dataset, before a train/test split, leak?
Fitting on all rows lets the future test rows influence the scaler's learned statistics. When those same rows are later used to score the model, the evaluation is no longer blind — that is the leak.
What does this print? make_pipeline names each step by lowercasing its class name. (Build-verified; the browser cannot run scikit-learn.)
from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) print(len(pipe.steps)) print(pipe.steps[0][0])
Two estimators were passed in, so there are two steps.
make_pipeline lowercases the class name for the step key, so StandardScaler becomes 'standardscaler'.
Write variance(xs) returning the population variance — the mean of the squared deviations from the mean. This is the quantity whose square root a StandardScaler divides by.
def variance(xs):
# mean of (x - mean) squared
pass
First compute the mean of xs.
Sum (x - mean) ** 2, then divide by len(xs).
def variance(xs):
mean = sum(xs) / len(xs)
return sum((x - mean) ** 2 for x in xs) / len(xs)
This centering step fits on the WHOLE list (train + test together), which leaks the test rows into the training mean. Fix it so the mean comes from the TRAIN portion only — the first n_train rows.
def scale_train_only(rows, n_train):
# BUG: fits the centering mean on every row, test rows included
mean = sum(rows) / len(rows)
return [r - mean for r in rows[:n_train]]
Slice the train rows first: train = rows[:n_train].
Compute the mean from train only, then subtract it from the train rows.
def scale_train_only(rows, n_train):
train = rows[:n_train]
mean = sum(train) / len(train)
return [r - mean for r in train]
You pass a make_pipeline(StandardScaler(), LogisticRegression()) to cross_val_score. What does this guarantee that manual scaling cannot?
Because the Pipeline is one estimator, cross_val_score rebuilds the whole scaler-model chain inside every fold, fitting the scaler on that fold's training rows alone. The validation rows never touch the scaler's fit, which is the structural guarantee manual code lacks.
Recap
- Leakage is an ordering bug: fitting preprocessing on all the data before splitting lets test rows shape the transform applied to them.
- A Pipeline binds the scaler and the model into one estimator that exposes the same
fit/predict/scoreverbs. - Scored by
cross_val_score, a Pipeline rebuilds the whole chain inside each fold, fitting the scaler on that fold's train rows alone. - The smell to hunt for is
fit_transformon the full dataset; the safe verbs arefiton train andtransformon the rest.
Next you will search over model settings — a hyperparameter search — and learn to read its results without fooling yourself, using a Pipeline so the search itself stays leakage-free.