You have trained a model that scores 94% on your test set, so you ship it to production — and watch it collapse to 62%. The data is the same, the code is the same, but the model has met real data for the first time and it is lost. This is the signature of data leakage: information from outside the training set sneaks into the model during training, giving it an unfair advantage that evaporates the moment the cheat sheet is taken away. The most common form of leakage is also the easiest to miss: preprocessing the entire dataset — scaling, imputation, or feature engineering — before you split it into train and test.
flowchart LR
A["raw data"] --> B{"split first?"}
B -->|"yes"| C["train stats"]
C --> D["scale train"]
D --> E["scale test with train stats"]
E --> F["honest score"]
B -->|"no"| G["scale all data together"]
G --> H["split"]
H --> I["inflated score"]
style F fill:#166534,color:#fff
style I fill:#b91c1c,color:#fff
What leaking looks like
Suppose you standard-scale an entire dataset before splitting. The global mean includes every test example, so when you subtract that mean from a training point, you are implicitly using test information to position it. The same happens with the global minimum and maximum in min-max scaling. The model does not see the test labels directly, but it sees the test distribution — where the centre is, how wide the spread is, and where the outliers sit. That is enough to let the model tune itself to patterns it should not know about, and the test score becomes a lie. The damage is invisible until production, because the test set was never truly unseen.
import math
def mean(data):
return sum(data) / len(data)
def std_dev(data):
m = mean(data)
return math.sqrt(sum((x - m) ** 2 for x in data) / len(data))
def standard_scale(data, m, s):
return [(x - m) / s for x in data]
train = [10, 12, 14, 16, 18]
test = [30, 32, 34]
m = mean(train)
s = std_dev(train)
scaled = standard_scale(test, m, s)
print("honest test scaled:", [round(x, 2) for x in scaled])
Run the honest pipeline and look at the test values. After scaling with the training mean and standard deviation, the first test point sits at 5.66 standard deviations above the training average. That is an honest signal: the test set is genuinely different from the training set, and the model should find it harder to predict. The large distance is not a bug; it is the truth. When you evaluate a model on these honestly scaled values, you are measuring how well it generalises to data that truly differs from what it has seen. That is the only measurement that matters.
How the statistics betray you
The honest pipeline scales the test set using the training mean and standard deviation. Because the test values are genuinely higher, they stay high after scaling — the first test point lands at 5.66 standard deviations above the training average. That honesty is what makes the evaluation realistic. The leaked pipeline computes the global mean across both sets, which is pulled upward by the test values. When you scale the test set with that global mean, its own bulk drags the centre toward itself, so the test points collapse to just one or two standard deviations away. They look normal, the model treats them as ordinary, and the score rises — but only because the test set was allowed to position the ruler.
flowchart TD A["test set: high values"] --> B["global mean rises"] B --> C["scale test with global mean"] C --> D["test points look normal"] D --> E["model score inflates"] style E fill:#b91c1c,color:#fff
import math
def mean(data):
return sum(data) / len(data)
def std_dev(data):
m = mean(data)
return math.sqrt(sum((x - m) ** 2 for x in data) / len(data))
def standard_scale(data, m, s):
return [(x - m) / s for x in data]
train = [10, 12, 14, 16, 18]
test = [30, 32, 34]
combined = train + test
m = mean(combined)
s = std_dev(combined)
scaled = standard_scale(test, m, s)
print("leaked test scaled:", [round(x, 2) for x in scaled])
Now run the leaked pipeline. The same test point collapses to just 1.02 standard deviations above average — a value that looks almost ordinary. The global mean, pulled upward by the test set itself, has camouflaged the real gap. If you trained a model on the leaked training data and tested it on these compressed test values, the test points would land in a region the model recognises, and the score would rise. But that score is a lie: the model has not generalised; it has been handed a cheat sheet encoded in the global statistics.
import math
train = [10, 20, 30]
test = [40]
mean_train = sum(train) / len(train)
std_train = math.sqrt(sum((x - mean_train) ** 2 for x in train) / len(train))
honest = (test[0] - mean_train) / std_train
print("honest:", round(honest, 2))
all_data = train + test
mean_global = sum(all_data) / len(all_data)
std_global = math.sqrt(sum((x - mean_global) ** 2 for x in all_data) / len(all_data))
leaked = (test[0] - mean_global) / std_global
print("leaked:", round(leaked, 2))
The fix: fit on train, transform on both
The honest pipeline has three steps. First, split your raw data into training and test sets with no peeking. Second, compute whatever statistics you need — mean, standard deviation, minimum, maximum — using only the training data. Third, apply those statistics to scale both the training set and the test set. The test set never influences the ruler; it is only measured by it. This rule extends beyond scaling to any preprocessing that learns from data: imputation, outlier clipping, and dimensionality reduction all follow the same pattern. If the step looks at the data to decide what to do, it must look at the training data only.
flowchart TD A["raw data"] --> B["train/test split"] B --> C["training data"] C --> D["compute statistics"] D --> E["scale train"] D --> F["scale test"] E --> G["model"] F --> H["evaluation"] style D fill:#166534,color:#fff style G fill:#3776ab,color:#fff
Beyond scaling
The 'fit on train, transform on both' rule applies to every preprocessing step that learns from data. Imputing missing values with the global mean leaks test information. Removing outliers by looking at the full dataset reshapes the training distribution based on test extremes. Even feature selection — choosing the top ten most correlated variables by looking at everything — leaks test labels into the model's architecture. The test set must remain unseen until the final evaluation. Treat it like a sealed exam paper: once opened, the grade is invalid.
What does this print? The train mean of [10, 20, 30] is used to scale the test value 40.
train = [10, 20, 30] test = [40] mean = sum(train) / len(train) scaled = (test[0] - mean) / 10 print(round(scaled, 1))
The train mean is 20.
(40 - 20) / 10 = 2.0
Write honest_scale(train, test) that standard-scales test using the mean and population standard deviation of train only. Return a list of scaled test values. If the standard deviation is zero, return a list of zeros.
import math
def honest_scale(train, test):
# scale test using train statistics only
pass
Compute the train mean first, then the train standard deviation.
Guard against s == 0 to avoid division by zero.
import math
def honest_scale(train, test):
m = sum(train) / len(train)
s = math.sqrt(sum((x - m) ** 2 for x in train) / len(train))
if s == 0:
return [0.0 for _ in test]
return [(x - m) / s for x in test]
This function leaks test data into the scaling step by using the global mean and standard deviation of the combined dataset. Fix it so the test set is scaled using only train statistics.
import math
def scale_test(train, test):
combined = train + test
m = sum(combined) / len(combined)
s = math.sqrt(sum((x - m) ** 2 for x in combined) / len(combined))
return [(x - m) / s for x in test]
Use sum(train) and len(train), not sum(combined) and len(combined).
Guard against s == 0 to avoid division by zero.
import math
def scale_test(train, test):
m = sum(train) / len(train)
s = math.sqrt(sum((x - m) ** 2 for x in train) / len(train))
if s == 0:
return [0.0 for _ in test]
return [(x - m) / s for x in test]
Why does scaling the entire dataset before train/test split inflate the test score?
When the global mean and standard deviation include test data, the test set drags the centre toward itself. After scaling, test points look closer to the training distribution than they actually are, so the model appears to generalise better than it does.
Write leakage_gap(train, test) that returns the difference between the mean of the test set after honest scaling (using train stats) and the mean of the test set after leaked scaling (using combined stats). Return the absolute difference as a float rounded to two decimal places. This quantifies how much the leakage compresses the test distribution.
import math
def leakage_gap(train, test):
# 1) honest: scale test with train mean and std
# 2) leaked: scale test with combined mean and std
# 3) return abs(honest_mean - leaked_mean), rounded to 2 decimals
pass
Compute honest scaled test values using only train statistics.
Compute leaked scaled test values using combined statistics.
Take the absolute difference of the two means and round to 2 decimals.
import math
def leakage_gap(train, test):
def mean(data):
return sum(data) / len(data)
def std(data):
m = mean(data)
return math.sqrt(sum((x - m) ** 2 for x in data) / len(data))
def scale(data, m, s):
if s == 0:
return [0.0 for _ in data]
return [(x - m) / s for x in data]
m_train = mean(train)
s_train = std(train)
honest = scale(test, m_train, s_train)
combined = train + test
m_global = mean(combined)
s_global = std(combined)
leaked = scale(test, m_global, s_global)
return round(abs(mean(honest) - mean(leaked)), 2)
Recap
- Data leakage is any situation where information from outside the training set influences model training. The most common form is preprocessing before splitting.
- Scaling before splitting lets test statistics shape the training pipeline. The test set pulls the global mean and standard deviation toward itself, so test points look more ordinary than they are.
- The honest pipeline is: split first, compute statistics on training data only, then apply those statistics to both sets. The test set is measured by the training ruler, never allowed to move it.
- This rule generalises to every preprocessing step that learns from data: imputation, outlier clipping, and feature selection must all follow the same pattern.
- A model that scores brilliantly on a leaked test set is not skilled — it is cheating. The score collapses in production because the cheat sheet is gone.