6.7 Which line should we fit?

Infinitely many lines pass through a scatter of points. Which one?

Every sample admits infinitely many possible lines. We therefore need a rule telling us which one deserves to be called the regression line.

A residual measures how badly a candidate line misses observation \(i\):

\[\hat{u}_i = y_i - \hat{y}_i = y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i\]

We want the residuals small. The obvious criterion is to make their sum as close to zero as possible.

That criterion fails, and it is worth seeing why.

Consider two lines. The first misses four points by \(+1, -1, +1, -1\). The second misses them by \(+8, -8, +8, -8\).

Both have residuals summing to exactly zero. By that criterion the two lines are equally good, which is absurd — the second is dreadful.

The problem is that positive and negative misses cancel. A criterion that lets errors offset one another can be satisfied by a line that is nowhere near the data.

The fix is the one from Section 1.2: square the deviations first, so that they cannot cancel and large misses count for more than small ones. We choose \(\hat{\beta}_0\) and \(\hat{\beta}_1\) to minimise

\[L = \sum_{i=1}^{n} \hat{u}_i^2 = \sum_{i=1}^{n}\left(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i\right)^2\]

This is the method of ordinary least squares, or OLS.

Notice that the criterion depends only on quantities we can observe. It is built from residuals, not errors — we could not minimise \(\sum u_i^2\) even if we wanted to, because nobody sees \(u\).

We now have a criterion. Finding the line that satisfies it is a matter of calculus, and it is where the next unit begins.

Regression chooses the line making the sum of squared vertical distances from the points to the line as small as possible.

Squaring does two things. It stops positive and negative misses from cancelling, and it makes one large miss count for more than several small ones.