A straight line is the simplest story you can tell about data: as the input rises, the output rises with it by a fixed amount, forever. That story is powerful when it happens to be true, and quietly misleading when it is not. Real relationships often curve — a price that falls and then rises, a growth rate that accelerates, a reaction rate that peaks and declines. When the truth bends and your model cannot, no amount of training will straighten out the fit. The line simply does the best straight job it can, and the best is not good enough.
That failure has a name: underfitting. The model is too simple to hold the pattern in the data, so its error stays high no matter how long you train. This lesson is about spotting underfitting, and about the one trick that resolves it — handing the model a way to bend.
flowchart LR A["data curves"] --> B["fit a straight line"] B --> C["the line cannot bend"] C --> D["systematic residuals<br/>low at the ends, high in the middle"] D --> E["underfitting: too simple"] style E fill:#b91c1c,color:#fff
What underfitting looks like
Underfitting is a property of the model's capacity, not of its training. A straight line carries two degrees of freedom — a slope and an intercept — so it can tilt and it can shift, but it cannot curve. Fit a line to data that genuinely arcs and the line cuts through the middle, floating above the data in some regions and sinking below it in others.
The clearest evidence lives in the residuals, the per-point prediction errors. If those errors scatter like random noise, the model has captured everything it reasonably can. If instead they trace a smooth shape — large at one end, small in the middle, large again at the other — the model is leaving real structure behind. That leftover structure is exactly the curve the line cannot reach.
xs = [0, 1, 2, 3, 4]
ys = [0, 1, 4, 9, 16] # a curve, not a straight line
n = len(xs)
mx = sum(xs) / n
my = sum(ys) / n
slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)
intercept = my - slope * mx
preds = [intercept + slope * x for x in xs]
residuals = [round(p - y, 3) for p, y in zip(preds, ys)]
mse = sum((p - y) ** 2 for p, y in zip(preds, ys)) / n
print('line: y =', round(intercept, 2), '+', round(slope, 2), '* x')
print('residuals:', residuals)
print('mse:', round(mse, 3))
Reading the residuals
The residuals come out negative at both ends and positive through the middle. That shape is the hidden curve itself, seen through the lens of the model's mistakes — the line drops below the rising data at the ends and rides above the dip in the middle. Genuine noise would scatter in sign and size; this error has a smooth, repeating form.
A model that underfits always leaves a fingerprint in its errors, and training longer will not erase it. The line is already the best possible line. The only fix is a model that can actually bend.
flowchart LR X["input x"] --> F["build features<br/>1, x, x squared"] F --> M["design matrix<br/>one row per example"] M --> E["same least-squares engine<br/>as the line fit"] E --> W["weights: bias, w1, w2"] style E fill:#10b981,color:#fff
The trick: let the model bend
The line struggles because its only tools are a slope and a shift. Give it one more tool — the ability to weigh the square of the input as well as the input itself — and it gains the power to curve. The model becomes prediction = bias + w1 * x + w2 * x squared, and fitting it means choosing three numbers instead of two.
Nothing about the fitting step changes. You add a column that holds the square of each input, treat it as a second feature, and solve exactly the same least-squares problem you solved for the plain line. The prediction is still a sum of terms; it simply draws on richer inputs now.
def solve(A, b):
n = len(A)
M = [list(A[i]) + [b[i]] for i in range(n)]
for c in range(n):
p = max(range(c, n), key=lambda r: abs(M[r][c]))
M[c], M[p] = M[p], M[c]
pv = M[c][c]
for r in range(n):
if r != c:
f = M[r][c] / pv
for k in range(c, n + 1):
M[r][k] -= f * M[c][k]
return [M[i][n] / M[i][i] for i in range(n)]
def least_squares(rows, targets):
cols = list(zip(*rows))
XtX = [[sum(cols[k][i] * rows[i][j] for i in range(len(rows)))
for j in range(len(cols))] for k in range(len(cols))]
Xty = [sum(cols[k][i] * targets[i] for i in range(len(rows))) for k in range(len(cols))]
return solve(XtX, Xty)
def design_row(x, degree):
return [x ** d for d in range(degree + 1)]
xs = [0, 1, 2, 3, 4]
ys = [0, 1, 4, 9, 16] # exactly the input squared
rows = [design_row(x, 2) for x in xs] # each row is [1, x, x squared]
w = least_squares(rows, ys)
preds = [sum(w[d] * row[d] for d in range(len(w))) for row in rows]
mse = sum((p - y) ** 2 for p, y in zip(preds, ys)) / len(ys)
print('weights:', [round(c, 3) for c in w])
print('mse:', round(mse, 6))
What the weights reveal
Fit a quadratic to data that came from a parabola and the weights land near a bias of zero, a weight on x of zero, and a weight on x squared of one — the model has rediscovered that the output equals the input squared. The very engine that drew a rigid line now reproduces a curve, because you handed it the right column to weigh.
Each weight is readable: the squared term's coefficient sets how sharply the curve bends, the linear term tilts it left or right, and the bias lifts or lowers the whole thing. Used well, capacity turns an unbending line into a flexible curve.
When bending goes too far
If a squared feature cleared up the underfit, why not add the cube, the fourth power, all the way up? Each new feature is more capacity, and capacity can only ever lower the training error — a polynomial of high enough degree can be driven through every single training point, pushing that error to zero.
That is not a win. A curve that stabs through every point is memorising the data, noise included, and it will whip wildly in the gaps between points where there is no data at all. Training error collapsing while real-world accuracy falls apart is called overfitting — the mirror image of this lesson. Underfitting is too little capacity; overfitting is too much.
flowchart TD C["model capacity<br/>polynomial degree"] --> L["degree 1: a line<br/>underfit, high bias"] C --> M["degree 2: a curve<br/>fits the trend"] C --> H["degree 10: wiggly<br/>overfit, high variance"] style L fill:#b91c1c,color:#fff style M fill:#166534,color:#fff style H fill:#b45309,color:#fff
def solve(A, b):
n = len(A)
M = [list(A[i]) + [b[i]] for i in range(n)]
for c in range(n):
p = max(range(c, n), key=lambda r: abs(M[r][c]))
M[c], M[p] = M[p], M[c]
pv = M[c][c]
for r in range(n):
if r != c:
f = M[r][c] / pv
for k in range(c, n + 1):
M[r][k] -= f * M[c][k]
return [M[i][n] / M[i][i] for i in range(n)]
def least_squares(rows, targets):
cols = list(zip(*rows))
XtX = [[sum(cols[k][i] * rows[i][j] for i in range(len(rows)))
for j in range(len(cols))] for k in range(len(cols))]
Xty = [sum(cols[k][i] * targets[i] for i in range(len(rows))) for k in range(len(cols))]
return solve(XtX, Xty)
xs = [0, 1, 2, 3, 4, 5]
ys = [0.3, 0.8, 4.4, 8.6, 16.3, 24.6] # a parabola plus a little noise
for degree in [1, 2, 5]:
rows = [[x ** d for d in range(degree + 1)] for x in xs]
w = least_squares(rows, ys)
preds = [sum(w[d] * row[d] for d in range(len(w))) for row in rows]
mse = sum((p - y) ** 2 for p, y in zip(preds, ys)) / len(ys)
biggest = max(abs(c) for c in w)
print('degree', degree, 'mse', round(mse, 3), 'largest weight', round(biggest, 1))
What does this print? A polynomial feature row holds the input raised to each power from 0 up to the degree.
def design_row(x, degree):
return [x ** d for d in range(degree + 1)]
print(design_row(3, 2))
x ** 0 is 1, x ** 1 is 3, x ** 2 is 9.
Python prints the list with brackets and comma-separated items.
Write design_matrix(xs, degree) that returns a list of rows, one per value in xs, where each row is [x**0, x**1, ..., x**degree]. So design_matrix([1, 2], 2) gives [[1, 1, 1], [1, 2, 4]].
def design_matrix(xs, degree):
# one row [x**0 ... x**degree] per x
pass
Use range(degree + 1) so you get powers 0 through degree.
A nested list comprehension does one row per x.
def design_matrix(xs, degree):
return [[x ** d for d in range(degree + 1)] for x in xs]
This design_row is meant to return [x**0, x**1, ..., x**degree], but it MULTIPLIES instead of exponentiating — so the constant column comes out 0 and the higher powers are wrong. Fix the one expression so each entry is x raised to the power d.
def design_row(x, degree):
return [x * d for d in range(degree + 1)] # bug: multiplies, not powers
x ** 0 should be 1, but x * 0 is 0.
Swap the * for **.
def design_row(x, degree):
return [x ** d for d in range(degree + 1)]
Write fit_line_mse(xs, ys) that fits the best straight line in closed form and returns its mean squared error, rounded to 4 decimals. Compute the slope as the sum of (x - mean_x) * (y - mean_y) divided by the sum of (x - mean_x) ** 2, the intercept as mean_y - slope * mean_x, then average the squared prediction errors.
def fit_line_mse(xs, ys):
# slope = cov(x,y) / var(x); intercept = my - slope*mx
# return round(mse, 4)
pass
mean_x and mean_y come from sum divided by n.
Predictions are intercept + slope * x; the error is their squared gap from y.
def fit_line_mse(xs, ys):
n = len(xs)
mx = sum(xs) / n
my = sum(ys) / n
num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
den = sum((x - mx) ** 2 for x in xs)
slope = num / den
intercept = my - slope * mx
preds = [intercept + slope * x for x in xs]
return round(sum((p - y) ** 2 for p, y in zip(preds, ys)) / n, 4)
You keep raising the polynomial degree. The training error falls all the way to zero, but a plot shows the curve whipping wildly between the data points. What is happening?
Near-zero training error with a wildly oscillating curve is the signature of overfitting. High capacity lets the model pass through every point, but it learns the noise rather than the trend, so it generalises poorly off-sample.
Recap
- A straight line underfits curved data — it lacks the capacity to bend, so the error stays high.
- The signature of underfitting is a systematic, smooth pattern in the residuals, not random scatter.
- The fix is feature engineering: add a column like x squared so the model can curve, then reuse the same least-squares engine.
- Polynomial regression is still linear regression — linear in the weights, not the features.
- Pushing capacity too far tips into overfitting: training error hits zero while the model memorises noise. That is the next lesson's problem.