The last lesson ended with a diagnosis: a model that swings too wildly has high variance, and the swing is caused by weights that grew large enough to chase the noise. Regularisation is the direct cure. Instead of letting the model pick whatever weights minimise training error, you change the rules so that large weights cost something. The model can still grow a weight large, but only if that growth pays for itself in a better fit. Idle weights that exist only to memorise one noisy point get squeezed back toward zero. The result is a simpler model — one that gives up a little training accuracy in exchange for a great deal more stability on new data. You trade a sliver of bias for a large cut in variance, and you come out ahead on the data you have not seen yet.
flowchart LR D["data-fit term<br/>how well predictions match"] --> L["total loss"] P["penalty term<br/>lambda times weight sizes"] --> L L --> M["minimise both:<br/>fit well, stay simple"] style P fill:#b45309,color:#fff style M fill:#166534,color:#fff
The L2 penalty
The most common penalty is L2 regularisation, also called ridge. It adds the sum of squared weights, scaled by a strength lambda, to the loss. The new quantity to minimise is the mean squared error plus lambda * sum(w squared). Every weight now has a running cost, and the bigger the weight, the steeper the cost. Squaring the weights means the penalty grows quickly: doubling a weight quadruples its penalty, so the model is punished hardest for its largest, most volatile weights — exactly the ones driving the variance. A weight has to earn its size by improving the fit enough to justify the tax. The lambda knob sets how strict the tax office is; turning it up forces every weight closer to zero.
def solve(A, b):
n = len(A)
M = [list(A[i]) + [b[i]] for i in range(n)]
for c in range(n):
p = max(range(c, n), key=lambda r: abs(M[r][c]))
M[c], M[p] = M[p], M[c]
pv = M[c][c]
for r in range(n):
if r != c:
f = M[r][c] / pv
for k in range(c, n + 1):
M[r][k] -= f * M[c][k]
return [M[i][n] / M[i][i] for i in range(n)]
def design(x, degree):
return [x ** d for d in range(degree + 1)]
def ridge_fit(xs, ys, degree, lam):
rows = [design(x, degree) for x in xs]
cols = list(zip(*rows))
XtX = [[sum(cols[k][i] * rows[i][j] for i in range(len(rows)))
for j in range(len(cols))] for k in range(len(cols))]
Xty = [sum(cols[k][i] * ys[i] for i in range(len(rows))) for k in range(len(cols))]
for j in range(1, len(XtX)):
XtX[j][j] += lam
return solve(XtX, Xty)
xs = [0, 1, 2, 3, 4]
ys = [0.5, 0.0, 2.5, 4.0, 8.5]
for lam in [0, 1, 10, 100]:
w = ridge_fit(xs, ys, 4, lam)
print("lambda", str(lam).ljust(4), "weights", [round(c, 2) for c in w])
Why the diagonal?
Ridge has an elegant closed form. Ordinary least squares solves XtX * w = Xty; ridge solves almost the same system, with lambda added to each diagonal entry of XtX first. That single change is the entire penalty — it nudges the matrix away from singularity and pulls every weight toward zero, harder as lambda grows. The code above adds lambda to the diagonal for every weight except the bias, because penalising the bias would stop the model from shifting up or down freely, which has nothing to do with complexity. Watch the printed weights collapse as lambda rises: at zero they are large and wild, at one hundred they are tiny. The model is being simplified in front of you.
flowchart TD A["lambda increases"] --> B["weights shrink toward zero"] B --> C["model grows simpler"] C --> D["variance falls,<br/>bias rises a little"] style D fill:#166534,color:#fff
def solve(A, b):
n = len(A)
M = [list(A[i]) + [b[i]] for i in range(n)]
for c in range(n):
p = max(range(c, n), key=lambda r: abs(M[r][c]))
M[c], M[p] = M[p], M[c]
pv = M[c][c]
for r in range(n):
if r != c:
f = M[r][c] / pv
for k in range(c, n + 1):
M[r][k] -= f * M[c][k]
return [M[i][n] / M[i][i] for i in range(n)]
def design(x, degree):
return [x ** d for d in range(degree + 1)]
def ridge_fit(xs, ys, degree, lam):
rows = [design(x, degree) for x in xs]
cols = list(zip(*rows))
XtX = [[sum(cols[k][i] * rows[i][j] for i in range(len(rows)))
for j in range(len(cols))] for k in range(len(cols))]
Xty = [sum(cols[k][i] * ys[i] for i in range(len(rows))) for k in range(len(cols))]
for j in range(1, len(XtX)):
XtX[j][j] += lam
return solve(XtX, Xty)
def predict(x, w):
return sum(wi * r for wi, r in zip(w, design(x, len(w) - 1)))
def mse(xs, ys, w):
return sum((predict(x, w) - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
xs_train = [0, 1, 2, 3, 4]
ys_train = [0.5, 0.0, 2.5, 4.0, 8.5]
xs_test = [5, 6]
ys_test = [12.5, 18.0]
print("lambda train_mse test_mse")
for lam in [0, 0.1, 1, 10]:
w = ridge_fit(xs_train, ys_train, 4, lam)
print(str(lam).ljust(7), round(mse(xs_train, ys_train, w), 2), " ", round(mse(xs_test, ys_test, w), 2))
Reading the lambda sweep
Run the sweep and two things happen at once. Training error rises monotonically, because the model is no longer allowed to chase every point. Test error does something more useful: it starts high (the unregularised degree-four fit is wildly overfit), drops to a minimum as the penalty tames the swing, then climbs again once the penalty crushes the model into an underfit constant. That second U-shape, this time traced against lambda, is how you choose the strength. Too little and you are back to overfitting; too much and you have underfit on purpose. The right lambda is the one that minimises held-out error, and the next lesson gives you the tool to estimate that error honestly instead of trusting a single lucky split.
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8] # the true relationship is y = 2x
def loss(w, lam):
data = sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
return data + lam * w ** 2
def gradient(w, lam):
data = (2 / len(xs)) * sum((w * x - y) * x for x, y in zip(xs, ys))
return data + 2 * lam * w
w = 0.0
for _ in range(300):
w = w - 0.01 * gradient(w, 1.0)
print("w with L2 =", round(w, 3), " loss =", round(loss(w, 1.0), 3))
print("without L2, w would head to 2.0; the penalty shrinks it below 2.0")
flowchart LR L["regularisation strength lambda"] --> S["too small: overfits<br/>weights stay large"] L --> J["just right: weights tamed"] L --> H["too large: underfits<br/>weights crushed to zero"] style S fill:#b91c1c,color:#fff style J fill:#166534,color:#fff style H fill:#b45309,color:#fff
What does this print? The L2 penalty is lambda times the sum of squared weights.
w = [3.0, 4.0] lam = 0.5 print(round(lam * sum(wi ** 2 for wi in w), 2))
3 squared is 9, 4 squared is 16, sum is 25.
Multiply 25 by 0.5.
Write l2_penalty(weights, lam) that returns lam * sum(w ** 2 for w in weights) — the penalty an L2-regularised model pays for its current weights.
def l2_penalty(weights, lam):
# lambda times the sum of squared weights
pass
Square each weight, sum them, then multiply by lambda.
The sign of a weight does not matter after squaring.
def l2_penalty(weights, lam):
return lam * sum(w ** 2 for w in weights)
Write ridge_gradient(data_grads, weights, lam) that returns the per-weight gradient of an L2-regularised loss. Each weight's gradient is its data gradient PLUS 2 * lam * weight. Return a list of the same length.
def ridge_gradient(data_grads, weights, lam):
# data_grad + 2 * lam * weight, element by element
pass
Pair each data gradient with its weight using zip.
Add 2 * lam * weight to each data gradient.
def ridge_gradient(data_grads, weights, lam):
return [g + 2 * lam * w for g, w in zip(data_grads, weights)]
This ridge function adds the penalty to EVERY diagonal entry, including the bias at index 0. But the convention is to leave the bias unpenalised so the model can still shift up and down freely. Fix it so it skips the bias (start the loop at index 1).
def add_penalty(XtX, lam):
for j in range(len(XtX)):
XtX[j][j] += lam
return XtX
range(len(XtX)) starts at 0, which includes the bias.
Start the range at 1 so index 0 is left untouched.
def add_penalty(XtX, lam):
for j in range(1, len(XtX)):
XtX[j][j] += lam
return XtX
Write ridge_step(w, data_grad, lam, lr) that performs one L2-regularised gradient-descent update on a single weight and returns the new weight. The update is w - lr * (data_grad + 2 * lam * w) — the data gradient plus the penalty gradient, stepped downhill.
def ridge_step(w, data_grad, lam, lr):
# w - lr * (data_grad + 2 * lam * w)
pass
The penalty gradient for one weight is 2 * lam * w.
Step downhill: subtract the learning rate times the total gradient.
def ridge_step(w, data_grad, lam, lr):
return w - lr * (data_grad + 2 * lam * w)
Recap
- Regularisation adds a penalty for large weights to the loss, so a weight must earn its size by improving the fit.
- L2 (ridge) penalises the sum of squared weights; in closed form it is ordinary least squares with
lambdaadded to the diagonal ofXtX. - As
lambdagrows, weights shrink toward zero, variance falls, and bias rises a little — you trade a sliver of one for a large cut of the other. - Sweeping
lambdatraces a U-shape in test error; the strength that minimises held-out error is the one to use. - L1 (lasso) uses absolute values and can drive weights to exactly zero, doubling as a feature selector.
- The bias term is left unpenalised so the model can shift freely.
- Choosing
lambdaneeds an honest estimate of held-out error — the subject of the next lesson, cross-validation.