You split a medical dataset at random and the model aces its test set — then collapses on a real new patient. The reason is structural: a single patient often contributes many rows (visits, scans, lab tests), and a random split scatters those rows across both sides. The model is not learning what makes people sick; it is learning to recognise individual patients, then being tested on more rows from those same patients. That is a leak, as real as scaling before the split, and it inflates the score for exactly the same reason: the test set was never truly unseen. In production the model meets strangers, not familiar faces, and the memorised hints are worthless.
flowchart LR P["patient A"] --> R1["row 1"] P --> R2["row 2"] P --> R3["row 3"] R1 --> TR["train"] R2 --> TE["test"] R3 --> TR TE --> X["model already met<br/>patient A in training"] style X fill:#b91c1c,color:#fff
What counts as a group
A group is any entity that contributes several rows: a patient with repeated visits, a customer with many orders, a device streaming readings, a hospital, or a store. Rows that share a group are correlated — they carry that entity's idiosyncrasies, from baseline vitals to purchasing habits. Random splitting assumes every row is independent of every other, an assumption grouped data breaks the moment it is collected. The remedy is to keep each group wholly on one side of the split, so the model is forced to generalise across entities rather than within them.
Why the score inflates
Suppose patient A appears in training with five rows and in the test set with two more. Anything patient-specific — their baseline vitals, the quirks of their case, even the handwriting on the intake form — has already been seen. A flexible model can encode 'this is patient A' implicitly and predict the test rows from that familiarity, not from any medically meaningful signal. The more rows per patient, the larger the leak, because each shared entity hands the model more identifying material. The reported accuracy rises, and it keeps rising as you collect more rows per patient, even though genuine skill at handling a brand-new patient has not improved at all.
# Leakage demo: rows of group 'A' appear in BOTH train and test.
train = {'A': 1, 'B': 1} # group -> times seen in training
test = ['A', 'A', 'C']
# A 'model' that only gets a row right if it saw the group in training
right = sum(1 for g in test if g in train)
print("leaked accuracy:", right, "/", len(test), "=", round(right/len(test), 2))
Now the group-aware split assigns every whole group to a single fold. When a fold becomes the test set, the model has trained on zero rows from any group inside it. The score it returns measures exactly what production demands: performance on entities the model has never encountered. The same logic governs customer churn, fraud detection across accounts, and fault prediction across machines — any time the production unit of prediction is an entity distinct from the ones in training. A model that aces grouped test data it has partly seen tells you nothing useful about the new entity that arrives tomorrow.
def group_kfold(groups, k):
uniq = sorted(set(groups))
fold_of = {g: i % k for i, g in enumerate(uniq)}
folds = [[] for _ in range(k)]
for i, g in enumerate(groups):
folds[fold_of[g]].append(i)
return folds
groups = ['A', 'A', 'B', 'B', 'C', 'C', 'D', 'D', 'E']
for fi, f in enumerate(group_kfold(groups, 3)):
print("fold", fi, f, "->", [groups[i] for i in f])
The other axis: time
Now order those same rows by when they happened — daily sales, hourly sensor logs, minute-by-minute prices. A random shuffle is happy to place last Tuesday in the training set and last Wednesday in the test set, or last month in training and this month in test. Two problems follow. Consecutive moments are strongly correlated, so the model trivially copies yesterday into today. More damning still, a random split can put a LATER time in training and an EARLIER time in test, letting the model train on the future to predict the past. Production only ever predicts forward, so a model tutored by the future is useless the day it ships.
flowchart LR T1["week 1"] --> T2["week 2"] T2 --> T3["week 3"] T3 --> T4["week 4"] T4 --> T5["week 5"] T5 --> TR["random split puts<br/>week 5 in TRAIN"] T2 --> TE["week 2 lands in TEST"] TE --> X["trained on the future<br/>to predict the past"] style X fill:#b91c1c,color:#fff style TR fill:#b45309,color:#fff
Peeking at the future is the deeper sin. Imagine training a stock model on weeks one through five and testing it on week three; the model has already seen the answer's neighbourhood. A high score here proves nothing about next week, because next week is, by definition, unwritten. The fix is causal: at every evaluation, the training window must end before the test window begins. No row whose timestamp falls inside the test period may appear in training, not even one that happened to be shuffled earlier.
def time_splits(n, train_size, step):
splits = []
start = 0
while start + train_size + step <= n:
tr_end = start + train_size
te_end = min(tr_end + step, n)
splits.append((list(range(start, tr_end)), list(range(tr_end, te_end))))
start += step
return splits
for tr, te in time_splits(10, 4, 2):
print("train", tr, "test", te)
The forward-chaining split enforces that causal order. It trains on the first window, tests on the next, then rolls both forward — training on a growing past, always testing on the immediate future. Each evaluation mirrors how you will actually use the model: standing at a moment in time, predicting what comes next, with only the past in hand. Averaging the scores across the rolling windows yields an estimate of forward prediction that a random split could never give you, because a random split was never respecting the arrow of time in the first place.
flowchart TD A["split 1:<br/>train weeks 1-2, test week 3"] --> B["split 2:<br/>train weeks 1-3, test week 4"] B --> C["split 3:<br/>train weeks 1-4, test week 5"] C --> D["train only on the past,<br/>test on the next future window"] style D fill:#166534,color:#fff
What does this print? Groups are assigned to folds round-robin by sorted name.
def fold_of_group(groups, k):
uniq = sorted(set(groups))
return {g: i % k for i, g in enumerate(uniq)}
print(fold_of_group(['A', 'A', 'B', 'C'], 2))
Sorted unique groups are A, B, C.
A is index 0 (fold 0), B is index 1 (fold 1), C is index 2 (fold 0).
Write group_split(groups, test_groups) returning (train_idx, test_idx): every row whose group is in test_groups goes to test, the rest to train.
def group_split(groups, test_groups):
# rows whose group is in test_groups -> test; the rest -> train
pass
Put test_groups into a set for fast membership checks.
Walk every row once and append its index to the right list.
def group_split(groups, test_groups):
test_set = set(test_groups)
train, test = [], []
for i, g in enumerate(groups):
(test if g in test_set else train).append(i)
return train, test
This time split trains on the FUTURE and tests on the PAST, which leaks tomorrow into today's prediction. Fix it so training uses the earlier rows and testing uses the later rows.
def time_split(n, train_size):
# BUG: trains on the FUTURE and tests on the PAST (leaks the future)
train = list(range(train_size, n))
test = list(range(0, train_size))
return train, test
Training rows must come BEFORE test rows in time.
train is range(0, train_size); test is range(train_size, n).
def time_split(n, train_size):
train = list(range(0, train_size))
test = list(range(train_size, n))
return train, test
Write group_kfold(groups, k) returning a list of k lists, each holding the row indices assigned to that fold. Assign groups to folds round-robin by sorted group name (the i-th sorted group goes to fold i % k), so no group appears in more than one fold.
def group_kfold(groups, k):
# sort unique groups, assign group i -> fold i % k, collect row indices
pass
Build fold_of as {group: i % k} over the sorted unique groups.
Make k empty lists, then drop each row index into the fold its group maps to.
def group_kfold(groups, k):
uniq = sorted(set(groups))
fold_of = {g: i % k for i, g in enumerate(uniq)}
folds = [[] for _ in range(k)]
for i, g in enumerate(groups):
folds[fold_of[g]].append(i)
return folds
You have 50,000 independent clicks from distinct anonymous users, with no meaningful timestamps and no shared grouping. Is a random train/test split appropriate?
Random splitting assumes rows are independent and identically distributed. When that genuinely holds — distinct users, no order that matters, no shared origin — a random split is correct, and group or time splits add nothing. Leakage appears only when rows share an entity or an order.
Recap
- Grouped data leaks under random splits. Rows from one patient, customer, or device land on both sides, letting the model memorise entities instead of learning signal.
- A group-aware split keeps each group wholly in one fold, so the test set holds only entities the model never saw — matching what production demands.
- Time-ordered data leaks too. A random split can train on the future and test on the past, which production never gets to do.
- Forward-chaining trains only on the past and tests on the next window, then rolls forward — the causal direction every real prediction must follow.
- When data has both a group and a time axis, respect both: split by time first, then keep groups from straddling the cut.