Residuals & Goodness of Fit (R-squared)
How far off is the line at each point — and how much of the story does it actually explain?
💡
The big idea: A regression line almost never passes through every point. The gap it leaves at each point — the residual — is where you find out whether a straight line was the right idea at all. Collect those gaps into a residual plot and it becomes a report card for the fit: patternless leftovers mean the line captured the trend, while a curve or a widening funnel means it did not. R-squared then turns the whole picture into a single number: the fraction of the variation in y that the line explains.
The leftover the line couldn't reach
You fit a straight line through a scatter of points. Almost none of the points land exactly on it — each one sits a little above or below. That vertical gap, from the point down (or up) to the line, is the residual.
Think of the line as the model's best guess and the residual as its miss. A point above the line has a positive residual (the line under-predicted); a point below has a negative one. Residuals are the raw material for answering two questions: was a straight line even the right shape, and how much of the pattern did it capture?
🔑
Residual = observed − predicted
For each point, the residual is the actual y-value minus the value the line predicted at that x. Write the prediction as \( \hat{y} \) ("y-hat"). So a residual of \( +2 \) means the real value came in 2 units above the line; a residual of \( -2 \) means 2 units below. The least-squares line is the one that makes these residuals, once squared and added up, as small as possible.
\[ e_i = y_i - \hat{y}_i \]
The residual for point i: its observed value minus the line's predicted value at that same x.
The residual plot: a report card for the line
Here is the trick that makes residuals so useful. Instead of eyeballing the scatterplot, plot the residuals themselves against x. If the line captured the real trend, the leftovers should be pure noise — scattered randomly above and below zero with no pattern.
A pattern in the residuals is the line confessing that it missed something:
- A curve (residuals swoop down then up, or up then down) means the true relationship bends — a straight line is the wrong shape.
- A funnel (residuals fan out, small on one side and large on the other) means the scatter around the line grows as x grows — the spread is not constant.
⚠️
A good fit shows nothing
The residual plot you want is boring: a shapeless, horizontal band of dots around zero. If you can describe a shape — a smile, a frown, a fan — the straight-line model has left structure on the table. A high correlation does not excuse a curved residual plot; the two checks answer different questions.
🎮 Interactive: residuals, SSE, and r-squared LIVE
Each blue point is joined to the least-squares line by a red residual segment. Switch the scatter shape and hit Regenerate: the noisy shape lengthens the residuals and drops r-squared, while a single outlier can swing the whole line and pull r-squared down. Watch SSE (the sum of squared residuals) and r-squared update, computed directly from the points.
✨
SSE adds up the squared misses
The sim reports SSE, the sum of squared residuals: square every red segment's length and add them. Squaring means big misses count far more than small ones, and it is exactly the quantity the least-squares line is built to minimize — no other straight line gives a smaller SSE for these points. Tighter points around the line means a smaller SSE; a lone outlier can inflate it dramatically.
R-squared: how much of the variation the line explains
SSE tells you the leftover error, but leftover compared to what? Compare it to how much y varied in the first place. Before drawing any line, the total spread in y is the sum of squared distances from the mean \( \bar{y} \); call it SST (total variation). The line removes some of that; whatever is left is the SSE.
The fraction of the variation the line explains is what remains after you subtract the leftover share:
\[ r^2 = 1 - \frac{\text{SSE}}{\text{SST}} = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2} \]
R-squared: one minus the share of y's variation left unexplained. It runs from 0 (the line explains nothing) to 1 (every point on the line).
✨
Why it is called r-squared
For a simple straight-line fit, this quantity is exactly the correlation \( r \) squared — hence the name. So \( r = 0.8 \) gives \( r^2 = 0.64 \): about 64% of the variation in y is explained by the linear relationship with x. Note that squaring erases the sign, so \( r = -0.8 \) also gives \( r^2 = 0.64 \). And honestly: \( r^2 \) measures only how well a straight line fits, and a strong linear association is still not evidence that x causes y.
📝 Worked example: Four points are (1, 6), (2, 7), (3, 12), (4, 13), and the least-squares line works out to ŷ = 2 + 3x. Find the residuals, the SSE, and r-squared, then interpret it.
- Predict at each x with \( \hat{y} = 2 + 3x \): at x = 1, 2, 3, 4 the line gives \( \hat{y} = 5, 8, 11, 14 \).
- Residual = observed − predicted: \( 6-5=+1 \), \( 7-8=-1 \), \( 12-11=+1 \), \( 13-14=-1 \). The signs alternate and sum to zero, as residuals from a least-squares line always do.
- SSE = sum of squared residuals \( = 1^2 + (-1)^2 + 1^2 + (-1)^2 = 4 \).
- Total variation SST uses the mean \( \bar{y} = (6+7+12+13)/4 = 9.5 \): deviations are \( -3.5, -2.5, 2.5, 3.5 \), so \( \text{SST} = 12.25 + 6.25 + 6.25 + 12.25 = 37 \).
- \( r^2 = 1 - \text{SSE}/\text{SST} = 1 - 4/37 \approx 0.89 \).
✓ The residuals are +1, −1, +1, −1; SSE = 4 and r² ≈ 0.89, meaning about 89% of the variation in y is explained by the straight-line relationship with x, leaving roughly 11% as leftover scatter.
Check your understanding
1. At a certain x, a data point has y = 20 and the regression line predicts ŷ = 17. The residual at that point is:
Residual = observed − predicted = 20 − 17 = +3. The point sits 3 units above the line, so the line under-predicted it.
2. You plot the residuals against x and see a clear U-shaped (curved) pattern. The best conclusion is:
A patternless residual plot signals a good linear fit; a systematic curve means the true relationship bends, so a straight line is the wrong model — even if r looks fairly high.
3. A simple linear regression reports r² = 0.64. The best interpretation is:
r² is the fraction of the variation in y explained by the line. It is not a count of points on the line nor a percent-correct — and here the correlation would be r = ±0.8, since 0.8² = 0.64.
4. A scatterplot has correlation r = −0.5. For this straight-line fit, r² equals:
r² = (−0.5)² = 0.25. Squaring removes the sign, so r² is never negative; about 25% of the variation in y is explained despite the negative direction of the association.
✅ Key takeaways
- A residual is observed minus predicted, e = y − ŷ: positive above the line, negative below.
- A residual plot is the fit's report card — a patternless band means a straight line fits; a curve means the wrong shape and a funnel means non-constant spread.
- SSE, the sum of squared residuals, is the total squared miss that the least-squares line makes as small as possible.
- r² = 1 − SSE/SST is the fraction of the variation in y explained by the line, running from 0 to 1.
- For a simple linear fit r² is just the correlation r squared, so it ignores sign — and it measures linear fit only, never causation.