A team spends three weeks tuning a neural network — more layers, more epochs, a carefully decayed learning rate — and the validation loss barely moves. Another team spends an afternoon cleaning the labels and re-running the same simple model, and the loss drops by a third. The first team was spending compute on the wrong thing.
The single most common waste in applied machine learning is reaching for a bigger model before asking where the error actually comes from. If the labels are noisy, no amount of capacity can help — the noise is a ceiling the model can never punch through. If the data is sparse, more parameters just give the model more ways to overfit. Before you scale the model, you owe yourself a diagnosis: is the bottleneck the data, or the model?
flowchart TD Q["model not good enough?"] --> First["build a trivial baseline first"] First --> Compare["does the model beat the baseline?"] Compare -->|"no"| Model["the model is the bottleneck"] Compare -->|"yes, but test error high"| Data["check data quality and label noise"] Data --> Noise["near the noise floor? more capacity will not help"] style Model fill:#b45309,color:#fff style Data fill:#3776ab,color:#fff style Noise fill:#b91c1c,color:#fff
The three places compute gets burned
Wasted effort in ML tends to cluster in three spots. First, scaling the model when the data is the limit — adding parameters that can only fit noise. Second, fitting the noise — pouring capacity into memorising random label errors that no model can predict, which drops training loss while real error stays flat. Third, ignoring the simple baseline — building a deep model before checking whether five lines of code already solve the problem.
All three share a root cause: optimising a number you can see instead of the error you actually have. The defence is measurement before machinery — a baseline, a noise estimate, and a clean held-out set — because you cannot fix a bottleneck you have not located.
Always build the trivial baseline first
Before any real model, compute the baseline: the error of the laziest possible prediction. For predicting numbers, that means predicting the mean of the targets — one number for the whole dataset. It is dumb, it takes a line of code, and it is the bar everything else must clear.
A surprising number of fancy models never beat it, because nobody checked. The baseline turns 'we got 0.85' from a boast into a statement that means something: 0.85 compared to what? If the mean-baseline already scores 0.84, your model added almost nothing.
# Clean relationship is y = 2x, but the recorded labels are noisy.
y_observed = [3, 3, 7, 7, 11, 11] # noisy version of [2, 4, 6, 8, 10, 12]
# Trivial baseline: predict the mean of the observed labels.
mean_y = sum(y_observed) / len(y_observed)
baseline_mse = sum((mean_y - y) ** 2 for y in y_observed) / len(y_observed)
print('mean-baseline MSE:', round(baseline_mse, 2))
The noise floor: the ceiling you cannot break
Now fit a model that memorises every training point exactly — a lookup table keyed on x. Its training MSE is zero, which looks miraculous next to the baseline's 10.67. But the model has learned the noise, not the signal, so it cannot generalise: hand it a new x and it has no entry to return.
The recorded labels differ from the true values by random errors, and those errors are unpredictable by definition. No model — however deep — can predict them. That random error is the noise floor, and it caps how low any model's error can honestly go on this data. Below the floor there is nothing real left to learn.
xs = [1, 2, 3, 4, 5, 6]
y_observed = [3, 3, 7, 7, 11, 11] # noisy labels
# A lookup-table model that memorises each training x exactly.
table = dict(zip(xs, y_observed))
train_mse = sum((table[x] - y) ** 2 for x, y in zip(xs, y_observed)) / len(xs)
print('memorised training MSE:', round(train_mse, 2))
flowchart LR Train["training data with label noise"] --> Fit["big model fits every label"] Fit --> Zero["training MSE 0"] Zero --> New["new point: cannot predict the noise"] New --> Floor["error is floored at the noise level"] style Zero fill:#166534,color:#fff style Floor fill:#b91c1c,color:#fff
The lever is usually the data
When a model sits near the noise floor, more capacity is pure waste — it fits the training noise harder without ever reducing the real error. The high-leverage move is to attack the data: clean the mislabelled rows, collect more samples, or engineer a feature that actually carries the signal. Each of those lowers the floor itself, so every model you ever train benefits.
Spending a week labelling 500 rows more carefully often beats spending it GPU-shopping. The data improvement compounds across every model and every future experiment; the model tweak helps exactly one run. Reach for capacity only after the data has stopped being the bottleneck.
xs = [1, 2, 3, 4, 5, 6]
y_clean = [2, 4, 6, 8, 10, 12] # the true relationship, no label noise
# Mean baseline on the clean labels.
mean_y = sum(y_clean) / len(y_clean)
mse_baseline = sum((mean_y - y) ** 2 for y in y_clean) / len(y_clean)
# A single-weight model, y = w * x, fit by gradient descent (built in an earlier lesson).
w = 0.0
for _ in range(300):
grad = (2 / len(xs)) * sum((w * x - y) * x for x, y in zip(xs, y_clean))
w = w - 0.01 * grad
mse_model = sum((w * x - y) ** 2 for x, y in zip(xs, y_clean)) / len(xs)
print('clean-label baseline MSE:', round(mse_baseline, 2))
print('clean-label model MSE:', round(mse_model, 3))
print('learned weight:', round(w, 2))
A cheaper experiment beats a bigger one
Before buying more compute, run the cheapest experiment that could change your mind. Train the simple model on 10% of the data, then 25%, then 50%, and watch the error curve. If error falls steeply as data grows, more data is the investment that will pay — go collect it. If the curve is already flat at 10%, the model is under-fitting and capacity, or better features, is the lever. If training error is low but held-out error is high, you are over-fitting and need regularisation or more data, not more parameters.
A learning curve like this costs minutes and points directly at the bottleneck. It is almost always a better first move than a larger model, because it tells you which direction is worth spending on.
flowchart TD Step1["1. build a trivial baseline"] --> Step2["2. measure where the error comes from"] Step2 --> Step3["3. is it data or capacity?"] Step3 -->|"data"| FixData["clean labels, collect more rows"] Step3 -->|"capacity"| Scale["only now scale the model"] style FixData fill:#3776ab,color:#fff style Scale fill:#166534,color:#fff
What does this print? The mean-baseline MSE for these four values.
y = [10, 10, 10, 20] mean_y = sum(y) / len(y) mse = sum((mean_y - v) ** 2 for v in y) / len(y) print(round(mse, 2))
Mean of [10, 10, 10, 20] is 12.5.
Squared gaps: 6.25 three times and 56.25 once, averaged over 4.
Write mean_baseline_mse(values) that returns the MSE of predicting the mean of values for every entry.
def mean_baseline_mse(values):
# predict the mean, average the squared gaps
pass
Compute the mean once, then average (mean - v) squared over all v.
A constant list has zero baseline error.
def mean_baseline_mse(values):
mean = sum(values) / len(values)
return sum((mean - v) ** 2 for v in values) / len(values)
This improvement_pct(baseline_mse, model_mse) should report how much SMALLER the model error is than the baseline, as a percentage of the baseline. The sign is flipped: it returns negative when the model is actually better. Fix the subtraction order.
def improvement_pct(baseline_mse, model_mse):
return (model_mse - baseline_mse) / baseline_mse * 100
Improvement is how much error went away: baseline minus model.
Divide by the baseline so the result is a percentage of the starting error.
def improvement_pct(baseline_mse, model_mse):
return (baseline_mse - model_mse) / baseline_mse * 100
Your model's training error is near zero, but test error is high and the labels are known to be noisy. What is the highest-leverage next step?
Near-zero training error on noisy labels means the model is fitting noise. Capacity and extra epochs cannot reduce the noise floor; only better data can lower the ceiling.
Write diagnose(baseline_mse, model_mse, noise_floor) returning a verdict string. If the model does not beat the baseline, return 'need a better model'. Otherwise, if the model is at or below 1.05 times the noise floor, return 'at the noise floor: fix the data'. Otherwise return 'model is improving: keep tuning'.
def diagnose(baseline_mse, model_mse, noise_floor):
# model not beating baseline -> model is the bottleneck
# near the noise floor -> data is the bottleneck
pass
Check the model-versus-baseline comparison first.
Only if it beats the baseline, check whether it has reached the noise floor (model_mse <= noise_floor * 1.05).
def diagnose(baseline_mse, model_mse, noise_floor):
if model_mse >= baseline_mse:
return 'need a better model'
if model_mse <= noise_floor * 1.05:
return 'at the noise floor: fix the data'
return 'model is improving: keep tuning'
Recap
- Reach for a bigger model only after you have ruled out the data as the bottleneck — capacity is the most common place to waste compute.
- Always build the trivial baseline first (predict the mean for numbers) and require every model to beat it; a score means nothing without a comparison.
- The noise floor — random label error — caps how low any model's error can honestly go. No amount of capacity breaks it.
- Cleaner data lowers the floor itself, so the improvement compounds across every model you ever train, while a model tweak helps exactly one run.
This module showed how good models go bad — leakage, drift, spurious correlation, paradox, imbalance, and wasted compute. The thread through all of them is the same: trust no score until you know what it was measured against.