Almost every model in machine learning is trained by the same idea, and it is simpler than it sounds: start with a bad guess, measure how wrong it is, then nudge the guess in the direction that makes it a little less wrong — and repeat. That direction is given by the gradient, the slope of the error, and following it downhill is gradient descent.
It is the engine under linear regression, logistic regression, and every neural network. Libraries hide it behind .fit(), but the whole thing is a short loop you can write yourself. In this lesson you build it for the simplest case — fitting a single weight — so that when you later call an optimiser, you know exactly what it is doing and why it sometimes fails.
The loss: how wrong are we?
Before you can go downhill you need to know which way is down. That is what a loss function gives you: a single number for how badly the current model fits the data. For predicting numbers, the standard choice is mean squared error (MSE) — the average of the squared differences between predictions and true values.
Squaring does two jobs: it makes every error positive (so they cannot cancel out), and it punishes big misses far more than small ones. A model that is off by 4 on one point contributes 16, while four points off by 1 each contribute only 4 in total. Minimising MSE therefore pulls hardest against the worst mistakes.
def mse(predictions, targets):
total = 0.0
for p, t in zip(predictions, targets):
total += (p - t) ** 2
return total / len(predictions)
print(round(mse([2, 4, 6], [2, 4, 6]), 2)) # perfect fit
print(round(mse([2, 4, 6], [3, 4, 5]), 2)) # off by 1 on two points
flowchart LR A["guess a weight"] --> B["measure loss"] B --> C["compute gradient<br/>(slope of loss)"] C --> D["step opposite the slope"] D --> B B --> E["loss stops falling:<br/>stop"] style C fill:#10b981,color:#fff style E fill:#166534,color:#fff
The gradient: which way is downhill?
Model a line through the origin, prediction = w * x, with one weight w. The loss is a function of w: pick a w, get a number. If you plotted loss against w you would see a bowl, and training means rolling to the bottom of that bowl.
The gradient is the slope of that bowl at your current w — it tells you which way the loss rises. For MSE the slope works out to (2/n) * sum((w*x - y) * x) over the data. You do not need to derive it here; you need to know its sign points uphill, so to go down you step in the opposite direction.
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8] # the true relationship is y = 2x
def loss(w):
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
def gradient(w):
return (2 / len(xs)) * sum((w * x - y) * x for x, y in zip(xs, ys))
w = 0.0
lr = 0.01
print('start w =', round(w, 3), 'loss =', round(loss(w), 3))
w = w - lr * gradient(w)
print('step 1 w =', round(w, 3), 'loss =', round(loss(w), 3))
The update rule
The single line that trains the model is the update rule: w = w - learning_rate * gradient. The gradient sets the direction; the learning rate sets how big a step you take. Subtracting moves you against the uphill slope — that is, downhill.
Run that line over and over and w walks toward the value that minimises the loss. Here the data was y = 2x, so a healthy loop should drive w toward 2 and the loss toward 0. Watching those two numbers move is how you tell training is working — and, later, how you diagnose it when it is not.
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8]
def loss(w):
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
def gradient(w):
return (2 / len(xs)) * sum((w * x - y) * x for x, y in zip(xs, ys))
w = 0.0
lr = 0.01
for step in range(1, 201):
w = w - lr * gradient(w)
if step % 50 == 0:
print('step', step, 'w =', round(w, 3), 'loss =', round(loss(w), 4))
flowchart TD S["learning rate"] --> A["too small:<br/>loss falls very slowly"] S --> B["just right:<br/>steady descent to the min"] S --> C["too big:<br/>overshoots, loss can grow"] style A fill:#b45309,color:#fff style B fill:#166534,color:#fff style C fill:#b91c1c,color:#fff
What does this print? MSE is the average of the squared differences.
def mse(predictions, targets):
total = 0.0
for p, t in zip(predictions, targets):
total += (p - t) ** 2
return total / len(predictions)
print(round(mse([1, 2], [3, 2]), 2))
First point: (1 - 3)^2 = 4. Second point: (2 - 2)^2 = 0.
Average of 4 and 0 is 2.0.
Write mse(predictions, targets) returning the mean of the squared differences between the two equal-length lists.
def mse(predictions, targets):
# average of (p - t) squared
pass
Sum (p - t) ** 2 across the pairs with zip.
Divide the total by len(predictions).
def mse(predictions, targets):
total = 0.0
for p, t in zip(predictions, targets):
total += (p - t) ** 2
return total / len(predictions)
Write predict(xs, w) that returns a list of predictions for the single-weight model prediction = w * x, one per value in xs.
def predict(xs, w):
# return [w * x for each x]
pass
Multiply each x by w.
A list comprehension does it in one line.
def predict(xs, w):
return [w * x for x in xs]
This update step ADDS the learning-rate times the gradient, so it walks UPHILL and the loss grows. Gradient descent must step AGAINST the slope. Fix the one operator so the weight moves toward a lower loss.
def step(w, grad, lr):
return w + lr * grad
The gradient points uphill; to descend you subtract it.
Change the + to a -.
def step(w, grad, lr):
return w - lr * grad
Write gradient(xs, ys, w) for the single-weight model, returning (2/n) * sum((w*x - y) * x) over the data. Then train(xs, ys, w, lr, steps) should apply the update rule w = w - lr * gradient(...) that many times and return the final w. xs and ys are equal-length lists.
def gradient(xs, ys, w):
# (2/n) * sum((w*x - y) * x)
pass
def train(xs, ys, w, lr, steps):
# apply w = w - lr * gradient(xs, ys, w), `steps` times
pass
At w = 2 the model is perfect, so the gradient is exactly 0.
train just loops the update rule; start w at whatever is passed in.
def gradient(xs, ys, w):
n = len(xs)
return (2 / n) * sum((w * x - y) * x for x, y in zip(xs, ys))
def train(xs, ys, w, lr, steps):
for _ in range(steps):
w = w - lr * gradient(xs, ys, w)
return w
Recap
- Gradient descent trains a model by repeatedly measuring the loss and stepping in the direction that lowers it.
- MSE — the mean of squared errors — is the loss for predicting numbers; squaring punishes big misses hardest.
- The gradient is the slope of the loss with respect to the weight; it points uphill, so you step the opposite way.
- The update rule is one line:
w = w - learning_rate * gradient. The gradient gives direction, the learning rate gives step size. - A too-large learning rate makes the loss diverge (
nan/infinity); a too-small one makes it crawl. Tuning it is a permanent part of the job.
You just wrote the core loop of nearly all of machine learning. Next you will extend it from one weight to many, and then to classification with logistic regression — the same engine, a different loss.