You ship a model that scores 92% in testing. Three months later the same model, on the same code, scores 71% — and nobody changed a line. The data changed underneath it. Data drift is the slow movement of the real world away from the world your model was trained on: customer tastes shift, sensors age, an economy reopens, a competitor cuts a price in half. The model learned a map of the old territory, and the territory has moved.
Drift is silent. Unlike a crash, it produces confident, wrong predictions, and the accuracy number only falls once enough damage is done. The defence is rarely a cleverer model; it is a detector that watches the inputs and sounds an alarm before the predictions rot.
flowchart LR M["boundary learned<br/>on world A"] --> A["world A: matches"] A --> G["good score"] M --> B["world B: shifted"] B --> D["score collapses"] style G fill:#166534,color:#fff style D fill:#b91c1c,color:#fff
Why a fixed boundary goes stale
Most models draw a boundary — a threshold, a line, a region — and then apply it forever after. A churn model might learn the rule predict churn when monthly usage drops below 5 gigabytes. That boundary was correct for the customers who existed at training time. If the whole market shifts upward, because a new default plan raises everyone's usage by six gigabytes, the very same customers who would have churned now sit at 11 gigabytes, safely above the line, and the model predicts fine for every one of them.
The model has not broken. Its boundary sits exactly where you set it. What moved is the distribution of the inputs, and a boundary drawn for one distribution is meaningless against another. That is the whole failure: the math is unchanged, and the predictions are confidently wrong.
THRESHOLD = 5 # midpoint of the class means in world A
def predict(x):
return 1 if x > THRESHOLD else 0
def accuracy(xs, ys):
correct = sum(1 for x, y in zip(xs, ys) if predict(x) == y)
return correct / len(xs)
# world A: what the boundary was drawn for
xs_A, ys_A = [2, 3, 8, 9], [0, 0, 1, 1]
# world B: every value shifted up by 6, labels unchanged
xs_B, ys_B = [8, 9, 14, 15], [0, 0, 1, 1]
print('accuracy on A:', accuracy(xs_A, ys_A))
print('accuracy on B:', accuracy(xs_B, ys_B))
flowchart TD T0["training time:<br/>boundary fits the data"] --> T1["later: the data shifts right"] T1 --> M["boundary now sits<br/>in the wrong place"] M --> W["confident wrong predictions"] style M fill:#b45309,color:#fff style W fill:#b91c1c,color:#fff
Run the boundary and watch the score halve. On world A the threshold of 5 separates the two classes perfectly, because the classes sit on either side of it. On world B the same values have slid upward by six: the class-0 points that used to sit at 2 and 3 now sit at 8 and 9, above the boundary, so the model labels them class 1 and is wrong. Half the predictions flip, and the accuracy falls from 100% to 50% without a single change to the model. That fall is data drift made visible.
import math
def mean(xs):
return sum(xs) / len(xs)
def std(xs):
m = mean(xs)
return math.sqrt(sum((x - m) ** 2 for x in xs) / len(xs))
def drift_score(reference, current):
return abs(mean(current) - mean(reference)) / std(reference)
reference = [1, 2, 3, 7, 8, 9] # the feature, at training time
stable = [2, 3, 7, 8] # later sample: same location
shifted = [8, 9, 14, 15] # later sample: moved far
print('stable drift: ', round(drift_score(reference, stable), 2))
print('shifted drift:', round(drift_score(reference, shifted), 2))
Turning a shift into a decision
A drift score only helps once you attach a threshold to it. A common reading, borrowed from production monitoring, treats a small shift as background noise, a moderate shift as a warning to watch closely, and a large shift as a reason to retrain. The exact cut-offs depend on how much error your application can stomach, but the shape is always the same: one number that summarises how far the world has moved, measured against a line you chose in advance.
Notice what the score does not look at. It watches the inputs, not the predictions and not the labels. That choice is deliberate. In production the true labels often arrive weeks late — by the time you know whether a prediction was right, the damage is already done. A drift check on the raw inputs can fire today, while there is still time to act.
flowchart LR
S["drift score"] --> L{"how large?"}
L -->|"small"| N["noise: keep watching"]
L -->|"moderate"| W["warn: investigate"]
L -->|"large"| R["retrain or retire"]
style N fill:#166534,color:#fff
style W fill:#b45309,color:#fff
style R fill:#b91c1c,color:#fff
A model trained in 2024 scores 92%. In 2026 the same code scores 70%, with no bug and no code change. What is the most likely cause?
When accuracy falls without any code change, the model has stayed fixed while the world moved. The inputs the model now meets belong to a different distribution than the one it was trained on, so its fixed boundary misfires.
What does this print? The boundary is fixed at 5; every value in the shifted set has moved above it.
THRESHOLD = 5 xs = [8, 9, 14, 15] ys = [0, 0, 1, 1] correct = sum(1 for x, y in zip(xs, ys) if (1 if x > THRESHOLD else 0) == y) print(correct / len(xs))
Only the two largest values (14 and 15) are above the threshold.
Their labels are 1, so 2 of 4 predictions are correct.
Write drift_score(reference, current) returning how many population standard deviations of reference its mean has moved: abs(mean(current) - mean(reference)) / std(reference). Use the population standard deviation (divide by n, not n-1).
import math
def drift_score(reference, current):
# abs(mean diff) / population std of reference
pass
Compute the reference mean, then its population standard deviation.
Return the absolute difference of the two means, divided by that std.
import math
def drift_score(reference, current):
def mean(xs):
return sum(xs) / len(xs)
m = mean(reference)
s = math.sqrt(sum((x - m) ** 2 for x in reference) / len(reference))
return abs(mean(current) - m) / s
Write classify_drift(reference, current) that returns "no drift" when the drift score is below 0.5, "watch" when it is below 1.0, and "retrain" otherwise. Reuse your population-std mean-shift score from the previous exercise.
import math
def classify_drift(reference, current):
# <0.5 -> "no drift", <1.0 -> "watch", else "retrain"
pass
Compute the same mean-shift score, then branch on it.
Use strict less-than so the boundaries land in the right zone.
import math
def classify_drift(reference, current):
def mean(xs):
return sum(xs) / len(xs)
m = mean(reference)
s = math.sqrt(sum((x - m) ** 2 for x in reference) / len(reference))
score = abs(mean(current) - m) / s
if score < 0.5:
return "no drift"
if score < 1.0:
return "watch"
return "retrain"
Why does a drift monitor watch the model's INPUT distributions rather than its prediction accuracy?
Accuracy needs the true label, which in many production systems is only known long after the prediction is made. Inputs arrive with every request, so a drift check on the inputs can raise an alarm while there is still time to react.
Recap
- Data drift is the input distribution moving away from the training distribution over time, so a model that scored well at launch silently degrades.
- A fixed boundary drawn for one distribution misfires on another: the data slides past the line, and confident wrong predictions result.
- Quantify the shift with a drift score — how many reference standard deviations the mean has moved — and compare it against thresholds you chose in advance.
- Watch the inputs, not the labels: inputs arrive immediately, while true labels can lag by weeks, by which point the damage is done.
- When drift strikes, diagnose what moved before you rebuild. A bigger model learns the same stale boundary; the fix is fresh data, a repaired sensor, or sometimes retirement.
Next you will meet a different kind of deception: a feature that correlates beautifully with the label for no causal reason at all.