Your fraud detector flags transactions all day, and at the end of the quarter it reports 99% accuracy. The team celebrates, ships it, and the finance lead asks a single question: of the 10 real fraud cases, how many did it actually catch? The answer is zero. The model learned the safest possible cheat — predict that nothing is fraud, ever — and because only 1 in 100 transactions is fraud, that lazy rule is right 99% of the time.
This is the imbalance trap. When one class dominates, accuracy stops measuring what you care about and starts measuring the class balance instead. A model can score beautifully on accuracy and be completely useless for the job it was built to do. The fix is not a fancier algorithm; it is a better yardstick, and the discipline of resampling without cheating.
flowchart TD Q["the four outcomes"] --> TP["TP, true positive"] Q --> FN["FN, false negative, a miss"] Q --> FP["FP, false positive, a false alarm"] Q --> TN["TN, true negative"] TP --> R["recall uses TP and FN"] FN --> R TP --> P["precision uses TP and FP"] FP --> P style R fill:#3776ab,color:#fff style P fill:#3776ab,color:#fff
The four outcomes, named
Every prediction falls into one of four boxes. A fraud case caught is a true positive; a fraud case missed is a false negative — a real loss. An honest transaction wrongly frozen is a false positive — a frustrated customer. An honest transaction correctly left alone is a true negative.
Accuracy lumps all four together and asks only 'was the prediction right?'. On imbalanced data that question is drowned out by the huge pile of true negatives, so the rare positives — the very things you built the model to find — vanish inside the average.
# 990 honest transactions, 10 fraud.
y_true = [0] * 990 + [1] * 10
y_pred = [0] * 1000 # predicts every transaction is honest
correct = sum(t == p for t, p in zip(y_true, y_pred))
print('accuracy:', round(correct / len(y_true), 3))
print('fraud caught:', sum(p == 1 for p in y_pred))
Accuracy tells a comforting lie
Run that block and the all-negative model prints accuracy 0.99 while catching zero fraud. The number is real, and it is meaningless, because the model never predicted a single positive. Accuracy rewards a model for agreeing with the majority, and on imbalanced data the majority is the easy, boring class. To judge a detector you need metrics that ignore the easy wins and focus on the rare class — precision and recall.
def confusion(y_true, y_pred, positive=1):
tp = sum(t == positive and p == positive for t, p in zip(y_true, y_pred))
fp = sum(t != positive and p == positive for t, p in zip(y_true, y_pred))
fn = sum(t == positive and p != positive for t, p in zip(y_true, y_pred))
return tp, fp, fn
def precision(y_true, y_pred, positive=1):
tp, fp, fn = confusion(y_true, y_pred, positive)
return tp / (tp + fp) if (tp + fp) else 0.0
def recall(y_true, y_pred, positive=1):
tp, fp, fn = confusion(y_true, y_pred, positive)
return tp / (tp + fn) if (tp + fn) else 0.0
# 990 honest, 10 fraud. Model catches 8 of the 10, but wrongly flags 20 honest.
y_true = [0] * 990 + [1] * 10
y_pred = [1] * 20 + [0] * 970 + [1] * 8 + [0] * 2
print('precision:', round(precision(y_true, y_pred), 3))
print('recall:', round(recall(y_true, y_pred), 3))
flowchart LR Data["990 negatives, 10 positives"] --> Model["model predicts all negative"] Model --> Acc["accuracy 0.99"] Model --> Rec["recall 0.0, caught none"] Acc --> Trap["accuracy hides the failure"] style Acc fill:#166534,color:#fff style Rec fill:#b91c1c,color:#fff style Trap fill:#b91c1c,color:#fff
Two numbers that refuse to be fooled
Recall asks: of all the real fraud, how much did we catch? Eight of ten gives 0.8 — the all-negative model's recall was 0.0, and no pile of easy true negatives can hide that. Precision asks: of everything we flagged as fraud, how much really was? Eight real catches out of twenty-eight flags gives 0.286 — the model cries wolf, freezing roughly three honest customers for every fraud it stops.
The two pull against each other. Flag more, and recall rises but precision falls; flag only the obvious cases, and precision rises but recall falls. You cannot maximise both at once, so the choice is a business decision — how much fraud can you tolerate missing, and how many honest customers can you afford to anger?
The metric your stakeholder actually wants
A fraud team and a marketing team read the same model differently. Fraud wants high recall — miss nothing, accept some false alarms, because one missed million-dollar fraud outweighs a hundred frozen accounts. A promotional-offer model wants high precision — every wrong offer costs money, so only flag the confident leads. There is no universally correct balance; there is the balance this business lives with. That is why a single accuracy number cannot even enter this conversation — it cannot express a trade-off.
The resampling trap
Fraud is rare, so a model trained on raw data barely sees any positives and learns little about them. The standard remedy is to resample — duplicate the rare fraud rows (oversampling) until the classes balance. Done right this helps. Done wrong, it is a leakage disaster.
The classic mistake is to oversample first, then split into train and test. Once you duplicate a fraud row before splitting, that row and its copy can land in different folds — so the test set contains a row the model already saw during training. The model memorises the answer and reports a wonderful score that evaporates the moment it meets genuinely unseen data. You have trained on the test set without realising it.
import random
random.seed(1)
# Six transactions: rows 0-3 honest, rows 4-5 fraud (the rare class).
rows = [0, 1, 2, 3, 4, 5]
labels = {4: 1, 5: 1}
# Honest order: SPLIT first, then oversample the rare class inside TRAIN only.
random.shuffle(rows)
train = sorted(rows[:4])
test = sorted(rows[4:])
positives_in_train = [r for r in train if labels.get(r, 0) == 1]
train_balanced = sorted(train + positives_in_train)
print('train:', train_balanced)
print('test:', test)
print('overlap:', sorted(set(train_balanced) & set(test)))
flowchart TD Wrong["oversample, then split"] --> Leak["a copy lands in test too, leakage"] Right["split, then oversample train"] --> Clean["test stays untouched, honest"] style Leak fill:#b91c1c,color:#fff style Clean fill:#166534,color:#fff
What does this print? Accuracy of a model that predicts 'honest' for every row, on 990 honest and 10 fraud.
y_true = [0]*990 + [1]*10 y_pred = [0]*1000 acc = sum(t == p for t, p in zip(y_true, y_pred)) / len(y_true) print(round(acc, 2))
Every prediction is 0. It matches all 990 honest rows and none of the 10 fraud.
990 correct out of 1000 = 0.99.
Write recall(y_true, y_pred, positive=1) returning TP / (TP + FN), or 0.0 if there are no real positives. TP is a true positive; FN is a positive the model missed.
def recall(y_true, y_pred, positive=1):
# TP / (TP + FN), or 0.0 if TP + FN == 0
pass
TP: actual and predicted both equal positive.
FN: actual equals positive but prediction does not. Divide TP by (TP + FN).
def recall(y_true, y_pred, positive=1):
tp = sum(t == positive and p == positive for t, p in zip(y_true, y_pred))
fn = sum(t == positive and p != positive for t, p in zip(y_true, y_pred))
return tp / (tp + fn) if (tp + fn) else 0.0
This precision divides by (tp + fn) — that is recall's denominator. Precision must divide by the count of everything predicted positive, which is (tp + fp). Fix the denominator (and the helper it sums).
def precision(y_true, y_pred, positive=1):
tp = sum(t == positive and p == positive for t, p in zip(y_true, y_pred))
fn = sum(t == positive and p != positive for t, p in zip(y_true, y_pred))
return tp / (tp + fn) if (tp + fn) else 0.0
Everything predicted positive is TP plus FP (actual negative, predicted positive).
Sum fp = (t != positive and p == positive), then divide tp by (tp + fp).
def precision(y_true, y_pred, positive=1):
tp = sum(t == positive and p == positive for t, p in zip(y_true, y_pred))
fp = sum(t != positive and p == positive for t, p in zip(y_true, y_pred))
return tp / (tp + fp) if (tp + fp) else 0.0
You duplicate the rare fraud rows to balance the classes, THEN split into train and test. What is the flaw?
Resampling before splitting lets a row and its copy land in different folds. The model then sees a copy of a test row during training — silent leakage that inflates every score on data that is not truly unseen.
Write resample_train(train, labels) that returns the training list with each positive row duplicated once. train is a list of row ids; labels maps every row id to its label (1 for the rare positive class). This is the honest oversampling step, applied to an already-split train fold.
def resample_train(train, labels):
# start from a copy of train, then append each positive row once more
pass
Build the result from list(train) so you do not mutate the input.
Loop over train; when labels[r] == 1, append r again.
def resample_train(train, labels):
out = list(train)
for r in train:
if labels[r] == 1:
out.append(r)
return out
Recap
- Imbalance makes accuracy a liar: a model that predicts only the majority class scores high while ignoring the rare class you actually care about.
- Recall is caught positives over all real positives; precision is real positives over everything flagged. Neither can be gamed by the easy majority.
- Resample to show the model more rare cases, but split first — oversample inside the training fold only, so no duplicated row reaches the test set.
- A great score on resampled data means nothing if the test set was contaminated. Honest evaluation beats optimistic numbers every time.
Next you will see where teams burn the most compute for the least gain — and why a bigger model is so often the wrong answer.