Gradient descent has only one knob a beginner truly fears, and it is not the weight or the data — it is the learning rate, the number that decides how big each step is. Set it well and the loss glides downhill to a fit; set it badly and the loop either crawls or explodes. The update rule itself is the same one-line formula from the previous lesson: w = w - learning_rate * gradient. The gradient picks the direction, and the learning rate picks how far to march in that direction. Everything that follows is about reading what the loss does after each step, because the loss is the honest scoreboard that tells you whether your rate is any good.
flowchart TD S["learning rate"] --> A["too small:<br/>loss barely falls"] S --> B["just right:<br/>steady descent to the min"] S --> C["too big:<br/>overshoots, loss grows"] style A fill:#b45309,color:#fff style B fill:#166534,color:#fff style C fill:#b91c1c,color:#fff
Three regimes, three symptoms
Vary the learning rate on the same problem and you will meet exactly three behaviours. Too small and the loss falls, but so slowly that hundreds of steps barely move it — the model is crawling, taking far longer than it should to reach the same answer. Just right and the loss drops steadily, levelling off as the weight settles at the minimum. Too big and each step overshoots the bottom of the bowl, landing higher up the far side than where it started, so the loss grows instead of shrinks — and a few steps later it has blown up to inf or nan.
The names for these are crawling, healthy convergence, and divergence. They are not subtle feelings; each leaves a clear trace in the loss over time, so the first debugging habit is to print the loss every few steps and read its shape.
def gradient(xs, ys, w):
n = len(xs)
return (2 / n) * sum((w * x - y) * x for x, y in zip(xs, ys))
def loss(xs, ys, w):
n = len(xs)
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / n
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8]
w = 0.0
lr = 0.02
for step in range(1, 21):
w = w - lr * gradient(xs, ys, w)
if step in (1, 5, 10, 20):
print('step', step, 'w =', round(w, 3), 'loss =', round(loss(xs, ys, w), 4))
Why a large rate diverges
Divergence feels like a bug but it is pure arithmetic. Picture the loss as a bowl, with your current weight a marble on one side. A sensible step rolls it a little way toward the bottom. A step that is too large flings the marble right past the bottom and up the opposite wall — and from higher up, the gradient is even steeper, so the next step is larger still. The misses compound: each overshoot lands further from the minimum than the last, and the numbers run away toward infinity.
The boundary between converging and diverging depends on how steep the bowl is, which your data's scale sets. That is why a rate that worked on one dataset can diverge on another, and why feature scaling matters — it reshapes the bowl. There is no universal correct rate; you discover it by watching the loss.
flowchart LR W["weight on one side"] --> O["step lands past<br/>the bottom"] O --> H["higher on the far side,<br/>steeper gradient"] H --> O2["next step is bigger"] O2 --> N["misses compound<br/>to infinity or nan"] style N fill:#b91c1c,color:#fff
def gradient(xs, ys, w):
n = len(xs)
return (2 / n) * sum((w * x - y) * x for x, y in zip(xs, ys))
def loss(xs, ys, w):
n = len(xs)
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / n
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8]
def run(lr, steps):
w = 0.0
for _ in range(steps):
w = w - lr * gradient(xs, ys, w)
final = loss(xs, ys, w)
if final != final or final == float('inf'):
return 'diverged'
return round(final, 3)
print('lr 0.05 ->', run(0.05, 100))
print('lr 0.0001 ->', run(0.0001, 50))
print('lr 0.5 ->', run(0.5, 30))
Diagnosing the regime in code
You do not have to read the numbers by eye; you can teach your loop to recognise its own behaviour. Track the loss as you step, and classify the run: if the loss ever becomes non-finite (inf or nan) or ends higher than it began, you have diverged and the rate is too large; if the loss falls to near zero, you have converged; if it falls only slightly after many steps, you are crawling and the rate is too small (or you simply need more steps). The final exercise asks you to write exactly this classifier.
A practical guard built on the same idea is to halt the moment the loss increases — there is rarely any reason to keep stepping once the loss is climbing, and stopping early prevents the explosive growth from wasting a long run. Drop the rate by an order of magnitude and start again.
Finding a rate that works
Because there is no closed-form answer, practitioners search on a log scale: start somewhere modest like 0.01, and if the loss falls healthily try tripling the rate until it diverges, then step back to the last value that held. The reason for log-scale jumps is that useful rates span whole orders of magnitude — 0.001, 0.01, 0.1 — while tiny nudges like 0.011 versus 0.012 rarely change the outcome. Once you have a rate that converges, you can decay it over time, taking smaller steps as you near the bottom so the weight settles instead of jittering around the minimum. This search-then-decay pattern is the skeleton of nearly every optimiser a library offers, and understanding it here makes those settings readable later.
flowchart TD L["the loss curve over time"] --> C["converging: falls and<br/>flattens near zero"] L --> D["diverging: ticks upward,<br/>then explodes"] L --> W["crawling: barely leaves<br/>its starting height"] style C fill:#166534,color:#fff style D fill:#b91c1c,color:#fff style W fill:#b45309,color:#fff
def gradient(xs, ys, w):
n = len(xs)
return (2 / n) * sum((w * x - y) * x for x, y in zip(xs, ys))
def loss(xs, ys, w):
n = len(xs)
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / n
xs = [1, 2, 3, 4]
ys = [2, 4, 6, 8]
def train_with_guard(lr, steps):
w = 0.0
prev = loss(xs, ys, w)
for step in range(1, steps + 1):
w = w - lr * gradient(xs, ys, w)
cur = loss(xs, ys, w)
if cur != cur or cur == float('inf') or cur > prev:
return 'halted at step ' + str(step) + ', loss rising'
prev = cur
return 'finished, loss = ' + str(round(cur, 3))
print('lr 0.02:', train_with_guard(0.02, 100))
print('lr 0.5 :', train_with_guard(0.5, 100))
What does this print? One update step: subtract the learning-rate-scaled gradient from the weight.
def update(w, grad, lr):
return w - lr * grad
print(update(2.0, 5.0, 0.1))
Compute lr * grad first: 0.1 * 5.0 = 0.5.
Then subtract from w: 2.0 - 0.5.
Write update(w, grad, lr) that performs one gradient-descent step and returns the new weight: subtract the learning-rate-scaled gradient from the current weight.
def update(w, grad, lr):
# w minus lr times grad
pass
Multiply the gradient by the learning rate before subtracting.
When the gradient is 0, the weight does not move.
def update(w, grad, lr):
return w - lr * grad
This step function forgets the learning rate entirely and subtracts the raw gradient — which is enormous, so it always overshoots. Fix it so the gradient is scaled by the learning rate first.
def step(w, grad, lr):
return w - grad
The gradient must be multiplied by lr before it is subtracted.
Without scaling, step(0.0, -3.0, 0.5) would jump to 3 instead of 1.5.
def step(w, grad, lr):
return w - lr * grad
Write diagnose(xs, ys, w0, lr, steps) that runs gradient descent for the single-weight model w * x starting at w0, then classifies the run: return "diverged" if the loss became non-finite or ended higher than it started, "converged" if the final loss fell below 1.0, otherwise "crawling". The gradient and loss helpers are provided.
import math
def gradient(xs, ys, w):
n = len(xs)
return (2 / n) * sum((w * x - y) * x for x, y in zip(xs, ys))
def loss(xs, ys, w):
n = len(xs)
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / n
def diagnose(xs, ys, w0, lr, steps):
# return 'converged', 'crawling', or 'diverged'
pass
Track the initial loss, then step the weight inside a loop the given number of times.
Classify at the end: non-finite or final-greater-than-initial means diverged; final below 1.0 means converged; otherwise crawling.
import math
def gradient(xs, ys, w):
n = len(xs)
return (2 / n) * sum((w * x - y) * x for x, y in zip(xs, ys))
def loss(xs, ys, w):
n = len(xs)
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / n
def diagnose(xs, ys, w0, lr, steps):
w = w0
initial = loss(xs, ys, w)
for _ in range(steps):
w = w - lr * gradient(xs, ys, w)
cur = loss(xs, ys, w)
if math.isnan(cur) or math.isinf(cur):
return "diverged"
final = loss(xs, ys, w)
if final > initial:
return "diverged"
if final < 1.0:
return "converged"
return "crawling"
After many steps, your loss has fallen only slightly from its starting value and is nowhere near zero. What is the most likely problem?
A loss that moves but barely, and never approaches zero, is the signature of a too-small rate: the steps are real but tiny, so convergence would take an impractical number of iterations. Raise the learning rate and the loss should fall faster.
Recap
- The learning rate is the single step-size control in
w = w - learning_rate * gradient; the gradient gives direction, the rate gives distance. - Three regimes: crawling (rate too small, loss barely moves), healthy convergence (steady fall to the minimum), and divergence (rate too large, overshoots compound to
nanorinf). - Divergence is arithmetic, not a bug — a step past the bottom lands on a steeper slope, so each miss is bigger than the last.
- The same rate can converge on one dataset and diverge on another because the loss surface's steepness depends on the data's scale.
- Diagnose by the loss curve, and guard your loops by halting the moment the loss starts to climb.
Next you will graduate from one weight to several — regression with multiple features — where the same update rule runs on a whole vector of weights at once.