This capstone does one thing: take a dataset from a raw table to a model you could honestly ship, and report metrics that would survive production. Nothing fancy - no deep learning, no leaderboard tricks. The whole point is that the boring, correct path is the one most teams skip. They scale before splitting and leak the test set into training. They tune on the test set until the number looks good, then watch it collapse on real users. They report accuracy on an imbalanced problem and ship a model that misses every important case.
By the end of this lesson you will have built the opposite: a leakage-free scikit-learn pipeline, cross-validated on the training data, evaluated on a test set touched exactly once, with calibrated probabilities and the right metric for the job. The code blocks below are real scikit-learn, verified at build time but not run in your browser; you will also run two short by-hand checks live to confirm the library agrees with the math you already built.
flowchart LR R["raw data"] --> S["split FIRST:<br/>train and test"] S --> P["build a Pipeline:<br/>scaler plus model"] P --> CV["cross-validate<br/>on TRAIN only"] CV --> T["evaluate on TEST<br/>once"] T --> CAL["calibrate<br/>probabilities"] style S fill:#3776ab,color:#fff style T fill:#b45309,color:#fff
The dataset
A small, honest table: twenty loan applications, each with an income, a debts column, and a label for whether the borrower defaulted. It is tiny on purpose - small enough to hold in your head, big enough that a model can find a real signal. The label is imbalanced: more people repay than default, which is exactly the situation where reporting plain accuracy misleads. Real projects start with a DataFrame like this, often from a CSV or a database query, and the first job is simply to look at it: how many rows, what the columns are, and how the labels are distributed. You cannot choose a metric, a split, or a model sensibly until you know the shape of what you are holding.
import pandas as pd
loans = pd.DataFrame({
"income": [25, 40, 60, 35, 80, 55, 30, 70, 45, 50,
20, 90, 38, 65, 42, 28, 75, 52, 33, 58],
"debts": [10, 5, 2, 12, 1, 4, 15, 3, 6, 8,
18, 0, 9, 2, 7, 14, 1, 5, 11, 4],
"default": [1, 0, 0, 1, 0, 0, 1, 0, 0, 0,
1, 0, 1, 0, 0, 1, 0, 0, 1, 0],
})
print(loans.shape)
print(loans["default"].value_counts().to_dict())
Split before you touch anything
The single most expensive mistake in applied ML is scaling or fitting anything on the full dataset before splitting. If the scaler sees the test rows, the test set's mean and spread leak into your preprocessing, and the model is gently trained on information it was supposed to be blind to. The test score then reads higher than the model deserves, and the gap shows up only in production. The fix is structural and simple: split the data into train and test FIRST, then never let the test set near a fit. Pass random_state so the split is reproducible, and stratify on the label so both halves keep the same class balance as the whole.
from sklearn.model_selection import train_test_split
X = loans[["income", "debts"]]
y = loans["default"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y,
)
print(len(X_train), len(X_test))
import random
random.seed(42)
indices = list(range(10))
random.shuffle(indices)
cut = len(indices) * 3 // 10 # hold out 30 percent
test_idx = sorted(indices[:cut])
train_idx = sorted(indices[cut:])
print("test", test_idx)
print("train size", len(train_idx), "test size", len(test_idx))
Build a Pipeline, not a sequence of steps
A model that needs scaled features is two operations: scale, then fit. Written as loose steps, it is easy to scale the training data, train the model, and then forget to scale the test data the same way before predicting - a training-serving skew bug that silently corrupts the score. scikit-learn's Pipeline bolts the steps into one object that scales and fits together, so the preprocessing is inseparable from the model. Cross-validation then runs the WHOLE pipeline on each fold, fitting the scaler on each training fold only - which is what makes leakage structurally impossible. You will not get that guarantee by chaining steps by hand; the Pipeline is the tool that earns it.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([
("scale", StandardScaler()),
("clf", LogisticRegression(random_state=42, max_iter=200)),
])
print(list(pipe.named_steps.keys()))
flowchart TD
subgraph BAD["leaky: scale then split"]
B1["fit scaler on ALL data"] --> B2["split into train and test"]
B2 --> B3["test rows influenced<br/>the scaler - inflated score"]
end
subgraph GOOD["honest: split then pipeline"]
G1["split first"] --> G2["pipeline scales<br/>each train fold only"]
G2 --> G3["test stays unseen"]
end
style B3 fill:#b91c1c,color:#fff
style G3 fill:#166534,color:#fff
Cross-validate on the training data only
With the pipeline built, cross-validation scores it honestly: it splits the TRAINING data into folds, and on each fold it fits the entire pipeline (scaler included) on the training portion and scores the held-out portion. The test set you set aside earlier is never involved. The result is a set of scores - one per fold - whose mean and spread tell you how the model performs on data it has not seen, with preprocessing leakage ruled out by construction. A wide spread across folds warns that the model is sensitive to exactly which rows it was trained on; a tight, high mean is your evidence it generalizes. Keep a fixed random_state on the splitter so the folds are reproducible.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, X_train, y_train, cv=5)
print("folds", len(scores))
print("mean", round(scores.mean(), 3))
Touch the test set exactly once
Everything so far used only the training data. The test set has been sitting untouched, which is exactly right. Now you fit the final pipeline on ALL the training data and score it on the test set once. That single number is your honest estimate of production performance - honest because the model and its preprocessing have never seen those rows. The temptation, once you see the number, is to go back and tune until it improves. Do not. Every time you peek at the test set and change the model in response, the test set stops being a test and becomes another training fold, and your honest estimate evaporates. Touch it once, report the number, and move on.
pipe.fit(X_train, y_train)
test_score = pipe.score(X_test, y_test)
print("test accuracy", round(test_score, 3))
def accuracy(predictions, truths):
correct = sum(1 for p, t in zip(predictions, truths) if p == t)
return correct / len(truths)
preds = [0, 1, 1, 0, 1]
truth = [0, 1, 0, 0, 1]
print(round(accuracy(preds, truth), 3))
Calibrate the probabilities
A model like logistic regression outputs a probability, and you want that probability to mean something real: if it says '0.8 chance of default' across a hundred loans, about eighty of them should actually default. Raw model probabilities often drift - systematically too confident, or too timid - which matters when a threshold turns the probability into a decision. Calibration maps the raw scores onto observed frequencies, usually by fitting a small model on the predictions. The accuracy of a hard 0/1 prediction and the trustworthiness of its probability are different properties; a model can be right most of the time and still have probabilities no one should act on. For any decision that thresholds on a probability - approve, flag, alert - calibration is not optional.
By hand, confirm it. Accuracy is the fraction of predictions that match the truth. What does this print?
def accuracy(predictions, truths):
correct = sum(1 for p, t in zip(predictions, truths) if p == t)
return correct / len(truths)
print(round(accuracy([0, 1, 1, 0, 1], [0, 1, 0, 0, 1]), 3))
Four of the five predictions match: indices 0, 1, 3, and 4.
4 divided by 5 is 0.8.
Split FIRST. A 30 percent stratified test split of ten rows. What are the test-set size and the classes present in it?
from sklearn.model_selection import train_test_split X = [[i] for i in range(10)] y = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1] Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y) print(len(Xte)) print(sorted(set(yte)))
30 percent of 10 rows is 3 held out for testing.
Stratifying keeps both classes represented, so 0 and 1 both appear.
A Pipeline is one object. What step names does this pipeline carry?
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([
("scale", StandardScaler()),
("clf", LogisticRegression(random_state=42, max_iter=200)),
])
print(list(pipe.named_steps.keys()))
named_steps keeps the names you gave each step, in order.
The first step was named 'scale', the second 'clf'.
Which pipeline leaks information from the test set into training?
Fitting the scaler on all data lets the test rows' mean and spread leak into preprocessing, inflating the test score. Split first and fit preprocessing inside the Pipeline on each training fold only, so the test set stays unseen.
Your production fraud model scores 95 of 100 non-fraud cases correctly but only 2 of 10 frauds. Overall accuracy looks high. Which single metric should you report to leadership to avoid misleading them?
On an imbalanced problem, accuracy hides the minority class. Recall (or precision, or F1) on the fraud class reveals that the model misses most frauds - the metric a paying business actually cares about. This is the accuracy trap from Module 5, now applied to a production decision.
Recap
- The honest path is split FIRST, then everything else - never fit or scale on the full dataset.
- A Pipeline bolts preprocessing to the model so scaling and fitting happen together, and cross-validation runs the whole pipeline per fold, making leakage structurally impossible.
- Cross-validate on the training data only to estimate generalization; the test set stays untouched.
- Touch the test set exactly once: fit on all the training data, score once, report. Tuning on the test set turns it into a training fold.
- Calibrate probabilities when a decision thresholds on them, and report the metric that matches an imbalanced problem - not the accuracy that flatters it.
You have now taken a dataset from a raw table to a model whose metrics would survive production, and you did it the boring, correct way. That is the whole course: the durable skill is not calling .fit(), it is building something you can trust.