Statistics › 📈 Relationships & Regression

Least-Squares Regression

Finding the one line that fits a scatter of points best — and reading what its slope and intercept actually mean.

Intro StatisticsAP Statistics level
💡
The big idea: A scatter of points rarely lines up perfectly, yet we can still draw the one straight line that fits best. Least-squares regression picks the line that makes the vertical misses — the residuals — as small as possible once you square them, giving a simple equation you can read to describe the data and use to predict.
🎯 By the end, you'll be able to
  • Explain what makes the least-squares line the best-fitting line: it minimizes the sum of the squared residuals
  • Compute the slope and intercept using b1 = r·sy/sx and b0 = ybar − b1·xbar
  • Interpret the slope as a rate of change and the intercept as the predicted value at x = 0
  • Use the fitted line to predict y from an x, and recognize when a prediction is unsafe (extrapolation)
📎 You should already know
  • Correlation and the coefficient r
  • Mean and standard deviation
  • Reading a scatterplot

Which straight line is the best one?

Plot two related measurements against each other and you usually get a cloud of dots that almost follows a straight line, but never exactly. So which straight line is the best one to draw through it? There are infinitely many lines to choose from, so we first need a rule that turns the vague word best into something we can actually compute.

For any candidate line and any point, there is a residual: the vertical gap between the point and the line, \(e_i = y_i - \hat y_i\). Points above the line have a positive residual, points below have a negative one. The least-squares rule says: square every residual, add them all up, and pick the line that makes that total as small as possible.

Why square? Squaring makes every miss positive, so the gaps above and below cannot cancel out, and it punishes big misses far more than small ones — so the winning line refuses to stray far from any point.

🔑 The least-squares regression line
The least-squares regression line \(\hat y = b_0 + b_1 x\) is the one line, out of all possible lines, that minimizes the sum of the squared residuals \(\sum (y_i - \hat y_i)^2\). We write the predicted value as \(\hat y\) (“y-hat”) to keep it distinct from an actual observed \(y\).

The slope and intercept have simple formulas

You do not have to test infinitely many lines by hand — calculus pins down the exact answer, and it depends only on things you already know: the correlation \(r\), the two standard deviations \(s_x\) and \(s_y\), and the two means \(\bar x\) and \(\bar y\).

The slope is just the correlation rescaled from standard-deviation units back into the real units of the data. The intercept is then fixed by forcing the line to pass through the point of averages \((\bar x, \bar y)\) — the least-squares line always goes through that balance point.

\[ \hat y = b_0 + b_1 x \]
The regression line: an intercept plus a slope times the predictor x. Feed in an x, read out a predicted y.
\[ b_1 = r\,\frac{s_y}{s_x} \qquad b_0 = \bar y - b_1\,\bar x \]
Slope from the correlation and the two spreads; intercept from the two means.
🎮 Interactive: fit a line and watch the residuals LIVE
Each blue dot is a data point, the purple line is the least-squares fit, and every red segment is a residual (point minus line). The readouts report the slope b1, intercept b0, r-squared, and SSE — the sum of squared residuals the line makes as small as possible. Switch the scatter to noisy, or add an outlier, and watch the line swing and the SSE change.
✨ The line is a balancing act
Follow the red residual segments in the tool. The least-squares line sits exactly where those squared gaps, added together (the SSE readout), are as small as they can possibly be — tilt or shift the line any other way and that total would grow. Now try the with an outlier shape: one faraway point has huge leverage, dragging the whole line toward itself and inflating the SSE, precisely because a big residual counts for its square.

Reading the slope, the intercept, and a prediction

Once you have \(\hat y = b_0 + b_1 x\), the two numbers tell the story:

  • The slope \(b_1\) is a rate: for each one-unit increase in \(x\), the predicted \(y\) changes by \(b_1\) units. A slope of 14 means the line predicts 14 more units of \(y\) for every extra unit of \(x\).
  • The intercept \(b_0\) is the predicted \(y\) when \(x = 0\). It carries a real-world meaning only when \(x = 0\) actually sits inside the range of your data; otherwise it is just where the line happens to cross the vertical axis.

To predict, drop an \(x\) into the equation and read off \(\hat y\). That is the whole job of a regression line: a rule that turns an \(x\) into a best-guess \(y\).

⚠️ What the line does not promise
  • Stay in range. A line fit on \(x\) values from 10 to 30 says nothing trustworthy about \(x = 100\). Predicting far outside the data is extrapolation, and it can be badly wrong.
  • A line is not a cause. A tight fit shows that \(x\) and \(y\) move together; it does not show that changing \(x\) makes \(y\) change. Correlation is not causation.
  • Straight lines only. Least squares fits the best straight line even when the true pattern is curved, so always look at the scatter, not just the numbers.
📝 Worked example: A study of homes records size x (in hundreds of square feet) and price y (in thousands of dollars). The data give xbar = 20, ybar = 300, sx = 5, sy = 80, and correlation r = 0.9. Find the least-squares line and use it to predict the price of a 2500-square-foot home.
  1. Slope: \(b_1 = r\,\dfrac{s_y}{s_x} = 0.9 \times \dfrac{80}{5} = 0.9 \times 16 = 14.4\).
  2. Intercept: \(b_0 = \bar y - b_1\,\bar x = 300 - 14.4 \times 20 = 300 - 288 = 12\).
  3. So the line is \(\hat y = 12 + 14.4\,x\). Reading the slope: each extra 100 sq ft is associated with about 14.4 thousand dollars more predicted price.
  4. A 2500-square-foot home means \(x = 25\): \(\hat y = 12 + 14.4 \times 25 = 12 + 360 = 372\).
✓ The least-squares line is yhat = 12 + 14.4x, predicting about $372,000 for a 2500-square-foot home. Note that x = 25 sits inside the data range, so the prediction is reasonable — but x = 0 is far outside it, so the intercept 12 is not a meaningful house price, just where the line meets the axis.

Check your understanding

1. Out of all possible straight lines, the least-squares regression line is the one that makes which quantity as small as possible?
By definition, least squares minimizes the sum of the squared residuals, ∑(y − yhat)². Squaring stops positive and negative misses from cancelling and penalizes large misses more.
2. A dataset has r = 0.5, sx = 4, and sy = 20. What is the slope b1 of the least-squares line?
b1 = r·sy/sx = 0.5 × 20 / 4 = 0.5 × 5 = 2.5.
3. In the fitted line yhat = 5 + 2x, what does the slope 2 tell you?
The slope is a rate of change: predicted y rises by b1 = 2 for every one-unit rise in x. The value at x = 0 is set by the intercept (here 5), not the slope.
4. A line is fit as yhat = 10 + 3x. What does it predict for x = 4?
Substitute x = 4: yhat = 10 + 3 × 4 = 10 + 12 = 22.
✅ Key takeaways
  • The least-squares regression line yhat = b0 + b1x is the single line that minimizes the sum of the squared residuals.
  • Slope: b1 = r·sy/sx, the correlation rescaled into the data's real units. Intercept: b0 = ybar − b1·xbar, which forces the line through the point of averages (xbar, ybar).
  • The slope is a rate: predicted y changes by b1 for each one-unit change in x.
  • The intercept is the predicted y at x = 0 — meaningful only when x = 0 lies within the data's range.
  • Predict by plugging an x into the equation, but avoid extrapolating far beyond the data, and remember a fitted line shows association, not causation.