Every model you train is pulled by two forces, and they work against each other. One pushes the model to capture the real pattern in your data; the other tempts it to capture the noise that just happens to be there. Get the balance wrong in either direction and the model fails, but for opposite reasons. Too little flexibility and the model cannot represent the truth, staying stubbornly wrong no matter how much data you feed it. Too much flexibility and the model fits every wiggle in your training set, including the wiggles that were pure accident, and then falls apart on new data. This is the central tension of machine learning, and it has a precise name: the bias-variance tradeoff. Naming it tells you not just why a model failed, but which kind of failure it is, and therefore which fix to reach for.
flowchart LR C["model capacity"] --> L["too simple<br/>high bias (underfit)"] C --> M["balanced<br/>lowest test error"] C --> H["too complex<br/>high variance (overfit)"] style L fill:#b91c1c,color:#fff style M fill:#166534,color:#fff style H fill:#b45309,color:#fff
Two failures, opposite causes
The two failure modes have names you will use for the rest of your career. Bias is the systematic error a model makes because its shape is too rigid for the truth — a straight line forced through curved data is biased, missing in the same direction every time. Variance is how much the model's predictions jump around when you train it on different samples drawn from the same underlying source. A model with high variance fits the noise as if it were signal, so each new training set produces a noticeably different model. These are not the same error measured two ways; they are different diseases. A high-bias model is consistently wrong; a high-variance model is inconsistently right. Most real models suffer from both at once, and the skill is knowing which one dominates.
def solve(A, b):
n = len(A)
M = [list(A[i]) + [b[i]] for i in range(n)]
for c in range(n):
p = max(range(c, n), key=lambda r: abs(M[r][c]))
M[c], M[p] = M[p], M[c]
pv = M[c][c]
for r in range(n):
if r != c:
f = M[r][c] / pv
for k in range(c, n + 1):
M[r][k] -= f * M[c][k]
return [M[i][n] / M[i][i] for i in range(n)]
def design(x, degree):
return [x ** d for d in range(degree + 1)]
def fit(xs, ys, degree):
rows = [design(x, degree) for x in xs]
cols = list(zip(*rows))
XtX = [[sum(cols[k][i] * rows[i][j] for i in range(len(rows)))
for j in range(len(cols))] for k in range(len(cols))]
Xty = [sum(cols[k][i] * ys[i] for i in range(len(rows))) for k in range(len(cols))]
return solve(XtX, Xty)
def predict(x, w):
return sum(wi * r for wi, r in zip(w, design(x, len(w) - 1)))
def mse(xs, ys, w):
return sum((predict(x, w) - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
xs_train = [0, 1, 2, 3, 4]
ys_train = [0.5, 0.0, 2.5, 4.0, 8.5]
xs_test = [5, 6]
ys_test = [12.5, 18.0]
print("degree train_mse test_mse")
for degree in [0, 1, 2, 4]:
w = fit(xs_train, ys_train, degree)
print(degree, " ", round(mse(xs_train, ys_train, w), 2), " ", round(mse(xs_test, ys_test, w), 2))
Reading the capacity ladder
Run the fit above. As the polynomial degree climbs, the training error keeps falling — at degree four it reaches zero, because a curve with five free parameters can be threaded through all five training points exactly. But the test error does something different: it drops, bottoms out near degree two, and then climbs again. That U-shape is the whole story. On the left, the model is too simple and both errors are high, which is underfitting driven by high bias. In the middle, the model captures the trend without chasing the noise. On the right, the training error is perfect and the test error is terrible, which is overfitting driven by high variance. The degree with the lowest training error is the worst possible choice; the degree with the lowest test error is the one you want.
flowchart TD E["total error<br/>on unseen data"] --> B["bias squared<br/>systematic miss"] E --> V["variance<br/>sensitivity to the sample"] E --> N["irreducible noise<br/>real-world randomness"] style B fill:#b45309,color:#fff style V fill:#3776ab,color:#fff style N fill:#6b7280,color:#fff
import random
def solve(A, b):
n = len(A)
M = [list(A[i]) + [b[i]] for i in range(n)]
for c in range(n):
p = max(range(c, n), key=lambda r: abs(M[r][c]))
M[c], M[p] = M[p], M[c]
pv = M[c][c]
for r in range(n):
if r != c:
f = M[r][c] / pv
for k in range(c, n + 1):
M[r][k] -= f * M[c][k]
return [M[i][n] / M[i][i] for i in range(n)]
def design(x, degree):
return [x ** d for d in range(degree + 1)]
def fit(xs, ys, degree):
rows = [design(x, degree) for x in xs]
cols = list(zip(*rows))
XtX = [[sum(cols[k][i] * rows[i][j] for i in range(len(rows)))
for j in range(len(cols))] for k in range(len(cols))]
Xty = [sum(cols[k][i] * ys[i] for i in range(len(rows))) for k in range(len(cols))]
return solve(XtX, Xty)
random.seed(7)
raw_x = list(range(12))
pool_y = [1.0 + 0.8 * x + random.uniform(-1.0, 1.0) for x in raw_x]
mx = sum(raw_x) / len(raw_x)
sx = (sum((x - mx) ** 2 for x in raw_x) / len(raw_x)) ** 0.5
pool_x = [(x - mx) / sx for x in raw_x]
def slopes(degree, seeds):
out = []
for s in seeds:
random.seed(s)
idx = random.sample(range(len(raw_x)), 5)
xs = [pool_x[i] for i in idx]
ys = [pool_y[i] for i in idx]
out.append(round(fit(xs, ys, degree)[1], 3))
return out
print("degree-1 slopes:", slopes(1, [1, 2, 3, 4]))
print("degree-4 slopes:", slopes(4, [1, 2, 3, 4]))
Watching variance directly
Bias shows up in a single fit; variance only shows up when you compare fits. The code above trains the same model on several different five-point samples drawn from one source and records the linear coefficient each time. The degree-one line barely moves — its slope hovers near one value across samples. The degree-four curve, fit through exactly five points, swings far more widely, because it has enough freedom to read each sample's particular noise as structure. That swing is variance, measured the only way that matters: how different is the model you would have built, had your data been slightly different? A model whose answer lurches when the data barely changes is not a model you can trust in production.
import random
TRUE = 10.0
random.seed(1)
def sample_mean(n):
return sum(random.gauss(TRUE, 3.0) for _ in range(n)) / n
trials = 4000
means = [sample_mean(5) for _ in range(trials)]
shrunk = [m * 0.9 for m in means]
def bias(est):
return sum(e for e in est) / len(est) - TRUE
def variance(est):
m = sum(est) / len(est)
return sum((e - m) ** 2 for e in est) / len(est)
print("plain mean : bias =", round(bias(means), 3), " variance =", round(variance(means), 3))
print("shrunk 0.9 : bias =", round(bias(shrunk), 3), " variance =", round(variance(shrunk), 3))
The decomposition, made visible
Total error on unseen data breaks into three pieces. Bias squared is the part the model gets wrong on average, even with infinite training sets. Variance is the part that comes from the model overreacting to the particular sample it saw. Irreducible noise is the real-world randomness no model can ever remove — the measurement error, the unmeasured variables, the genuine chaos in the system. The block you just ran shows the tradeoff in its purest form. A plain average is unbiased but wobbly; multiplying every estimate by 0.9 shrinks the wobble (variance falls to about 81% of its old value) at the cost of a small, steady bias. You cannot drive bias and variance to zero at the same time, because reducing one usually raises the other.
flowchart LR A["add flexibility"] --> B["bias falls"] A --> C["variance rises"] B --> D["sweet spot:<br/>neither error dominates"] C --> D style D fill:#166534,color:#fff
What does this print? MSE is the mean of the squared differences between predictions and targets.
def mse(preds, targets):
return sum((p - t) ** 2 for p, t in zip(preds, targets)) / len(preds)
print(round(mse([3.0, 3.0], [1.0, 5.0]), 2))
First point: (3 - 1)^2 = 4. Second point: (3 - 5)^2 = 4.
Average of 4 and 4 is 4.0.
Write error_gap(train_preds, train_targets, test_preds, test_targets) that returns the test mean squared error minus the train mean squared error. A large positive gap is the signature of overfitting. Each (preds, targets) pair has equal length.
def error_gap(train_preds, train_targets, test_preds, test_targets):
# test_mse - train_mse
pass
Define a small mse helper that averages the squared differences.
Return test mse minus train mse.
def error_gap(train_preds, train_targets, test_preds, test_targets):
def mse(preds, targets):
return sum((p - t) ** 2 for p, t in zip(preds, targets)) / len(preds)
return mse(test_preds, test_targets) - mse(train_preds, train_targets)
What does this print? The best model is the degree with the lowest TEST error, not the lowest training error.
test_errors = {"deg0": 12.5, "deg1": 5.0, "deg2": 1.2, "deg4": 18.7}
best = min(test_errors, key=lambda k: test_errors[k])
print(best)
Compare the test error values: 12.5, 5.0, 1.2, 18.7.
min with the key returns the degree whose value is smallest.
This function is meant to flag overfitting (a model that does far better on its training data than on unseen data), but its comparison is BACKWARDS — it calls a perfect-but-memorised model safe and an honest model overfit. Fix the one operator so it returns True exactly when test error is greater than train error.
def is_overfitting(train_mse, test_mse):
return train_mse > test_mse
Overfitting means the model does WORSE on test data than on train data.
So test error should be the larger side of the comparison.
def is_overfitting(train_mse, test_mse):
return test_mse > train_mse
Write estimator_variance(estimates) that returns the variance of a list of numbers: the mean of their squared deviations from their own mean. This measures how much an estimator's answer swings across different samples — i.e. its variance.
def estimator_variance(estimates):
# mean of (e - mean) squared
pass
First find the mean of the estimates.
Then average the squared distance of each estimate from that mean.
def estimator_variance(estimates):
m = sum(estimates) / len(estimates)
return sum((e - m) ** 2 for e in estimates) / len(estimates)
Recap
- Every model balances bias (systematic error from being too rigid) against variance (sensitivity to the particular training sample).
- Underfitting is high bias: both train and test error are high, because the model cannot represent the pattern.
- Overfitting is high variance: train error is near zero while test error stays high, because the model memorised the noise.
- The test-error curve against complexity is U-shaped; the degree in the middle, not the one with zero training error, is the one to ship.
- Total error splits into bias squared plus variance plus irreducible noise, and you cannot shrink bias and variance at the same time.
- Next you will tame variance directly, by penalising the large weights that cause the swing — a technique called regularisation.