Your churn model reports 99.7% accuracy, so you ship it — and within a week the business is furious. The model flags almost nobody, and the customers it does flag were already leaving. You did not build a model of churn. You built a model of a column that already contained the answer.
This is target leakage: a feature whose value is derived from the label, or from information that only exists after the outcome you are predicting. The model does not learn to predict; it learns to copy. Its score looks miraculous because, on every row including the test set, the answer is sitting in plain sight among the inputs.
flowchart LR R["row: features + label"] --> L["a leak column<br/>encodes the label"] L --> M["model reads the leak,<br/>not the real signal"] M --> S["near-perfect score<br/>(a mirage)"] style L fill:#b91c1c,color:#fff style S fill:#b91c1c,color:#fff
How the answer hides among the features
A leakage feature is rarely the label copied verbatim. More often it is a column computed from the label, or a timestamp that can only be known once the outcome has happened. A cancellation_date field in a churn dataset is leakage, because a cancellation date only exists for customers who have already churned. A refund_amount field in a fraud model is leakage for the same reason — you only learn the refund once the fraud has been resolved.
The model has no way to tell a cheat sheet from a genuine signal. It simply finds the feature that most reliably tracks the label, and a column derived from the label always wins that race. The damage stays invisible during training, because the column is present on every row, including the rows you held back for testing.
def class_means(rows, labels):
sums, counts = {}, {}
for x, y in zip(rows, labels):
sums[y] = sums.get(y, 0) + x
counts[y] = counts.get(y, 0) + 1
return {y: sums[y] / counts[y] for y in sums}
def predict(means, x):
best_y, best_d = None, None
for y, m in means.items():
d = abs(x - m)
if best_d is None or d < best_d:
best_d, best_y = d, y
return best_y
def accuracy(feat, labels):
means = class_means(feat, labels)
correct = sum(1 for x, y in zip(feat, labels) if predict(means, x) == y)
return correct / len(labels)
y = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]
signal = [2, 4, 1, 3, 5, 8, 6, 9, 7, 4] # a weak real feature
leak = y[:] # the answer itself
print('accuracy on signal:', accuracy(signal, y))
print('accuracy on leak: ', accuracy(leak, y))
Run it and the gap is total. The real signal barely separates the classes, so the centroid classifier lands at 80%. The leak column, which is the label, scores 100%. Nothing about the algorithm changed between the two runs — only which column it was allowed to read. That is the whole trick of target leakage: a feature that already contains the answer makes any model look brilliant, and the brilliance is worthless.
Why a held-out test set does not save you
The usual defence against an over-optimistic score is a train/test split: hold back rows the model never saw, and measure accuracy there. That defence fails against target leakage, because the leak column is filled in on the test rows too. The model never saw those particular rows, yet each one still carries the answer in its feature set, so copying the leak works on the test set as well as on the training set.
This is what makes leakage so dangerous: it survives the exact check you rely on to catch cheating. The only thing that exposes it is asking, for every feature, whether you would genuinely have its value at the moment you need to make a prediction.
def class_means(rows, labels):
sums, counts = {}, {}
for x, y in zip(rows, labels):
sums[y] = sums.get(y, 0) + x
counts[y] = counts.get(y, 0) + 1
return {y: sums[y] / counts[y] for y in sums}
def predict(means, x):
best_y, best_d = None, None
for y, m in means.items():
d = abs(x - m)
if best_d is None or d < best_d:
best_d, best_y = d, y
return best_y
y = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]
signal = [2, 4, 1, 3, 5, 8, 6, 9, 7, 4]
leak = y[:]
train_idx = [0, 1, 2, 5, 6, 7]
test_idx = [3, 4, 8, 9]
def score(feat):
means = class_means([feat[i] for i in train_idx], [y[i] for i in train_idx])
correct = sum(1 for i in test_idx if predict(means, feat[i]) == y[i])
return correct / len(test_idx)
print('held-out signal:', score(signal))
print('held-out leak: ', score(leak))
flowchart TD
S["train/test split"] --> T["test rows held back"]
T --> Q{"leak column<br/>present on test rows?"}
Q -->|"yes"| C["model still copies<br/>the answer"]
C --> P["100% on held-out test"]
Q -->|"no"| H["honest score"]
style P fill:#b91c1c,color:#fff
style H fill:#166534,color:#fff
labels = [0, 0, 0, 1, 1, 1]
leak = labels[:] # the answer, copied into a column
signal = [3, 1, 4, 8, 6, 9] # a real but imperfect feature
def match_rate(feature, labels):
hits = sum(1 for f, y in zip(feature, labels) if f == y)
return hits / len(labels)
print('leak matches label: ', round(match_rate(leak, labels), 2))
print('signal matches label:', round(match_rate(signal, labels), 2))
The prediction-time test
There is a single question that exposes every leak, and you can ask it of any feature before it enters the dataset: at the moment you need to make a prediction, will you actually know this value? If the answer is no, the feature is leakage, regardless of how well it scores.
A cancellation date is unknown for a customer who has not churned yet. A complication flag is unknown at the moment of admission. A column that summarises next month's transactions is unknown on the day a purchase is made. In each case the value only exists after the outcome, so a model that reads it is reading the future. The score such a feature buys you is not skill you can deploy; it is a countdown to embarrassment in production, the moment the column that held the answer is no longer there.
flowchart LR
F["a candidate feature"] --> G{"available at<br/>prediction time?"}
G -->|"yes"| K["keep it"]
G -->|"no"| D["drop it — it is leakage"]
style K fill:#166534,color:#fff
style D fill:#b91c1c,color:#fff
A model predicts, at the moment a loan application is submitted, whether the applicant will default in the next 12 months. Which column is target leakage?
Missed payments over the next 12 months can only be counted after the prediction window has passed, so the column encodes the outcome. Credit score, loan amount, and income are all known at application time and are legitimate features.
What does this print? Every test row's leak value equals its true label, so a model that copies the leak is never wrong.
leak = [0, 0, 1, 1] labels = [0, 0, 1, 1] correct = sum(1 for p, y in zip(leak, labels) if p == y) print(correct / len(labels))
Every leak value matches its label exactly.
4 matches out of 4 rows is 1.0.
Write match_rate(feature, labels) that returns the fraction of rows where feature[i] == labels[i]. This is exactly what a model exploiting a leak column does: it copies the feature as its prediction.
def match_rate(feature, labels):
# fraction of rows where feature[i] == labels[i]
pass
Count rows where the two lists agree, using zip.
Divide the count by the number of rows.
def match_rate(feature, labels):
hits = sum(1 for f, y in zip(feature, labels) if f == y)
return hits / len(labels)
Write most_suspicious(feature_to_values, labels) that returns the name of the feature whose values match labels most often. feature_to_values maps a feature name to its list of values (one per row); labels is the list of true labels. A feature that matches the label on almost every row is the classic smell of target leakage. On a tie, return the first feature you encountered.
def most_suspicious(feature_to_values, labels):
# return the feature name with the highest match rate vs labels
pass
Loop over the dict; for each feature compute its match rate against labels.
Track the best rate with a strict greater-than so the first feature wins ties.
def most_suspicious(feature_to_values, labels):
best_name, best_rate = None, -1.0
for name, values in feature_to_values.items():
rate = sum(1 for v, y in zip(values, labels) if v == y) / len(labels)
if rate > best_rate:
best_rate, best_name = rate, name
return best_name
A held-out test set is the standard guard against over-optimistic scores. Why does it fail to catch target leakage?
Target leakage is not caused by training peeking at test labels; it is caused by the answer being present in the feature set on every row, including held-out ones. A split cannot remove a column the rows themselves carry.
Recap
- Target leakage is a feature whose value comes from the label or from information only available after the outcome — a cheat sheet the model learns to copy.
- A leak column produces a near-perfect score that no real feature can match, because it contains the answer outright.
- It survives a train/test split: the held-out rows carry the leak too, so the impossible score persists on data the model never trained on.
- Catch every leak with one question: would you actually know this feature's value at prediction time? If not, drop it.
- This is distinct from preprocessing leakage (scaling before the split) — that is a pipeline-order bug; target leakage is a feature-design bug.
Next you will watch a model rot for a different reason — not because it cheated, but because the world moved on around it.