7.4 Why does OLS get the right answer on average?

The histogram was centred on the truth. Was that luck?

The histogram in Figure 7.4 had two striking features.

First, the slopes varied from sample to sample — sometimes a good deal.

Second, they fluctuated around the full-sample value rather than sitting systematically above or below it. The average of five thousand slopes was 0.190 against 0.189 from all 680 students.

Was that simply good fortune with this population, or does OLS always behave this way?

Showing that it always does requires assumptions, and this is a convenient place to name the set they belong to.

Regression rests on a standard list, known collectively as the assumptions of the classical linear regression model, or CLRM. There are six. Four of them deliver unbiasedness, and they are the four we need here.

CLRM1 — linear in parameters. The population model is \(y = \beta_0 + \beta_1 x + u\).

CLRM2 — random sampling. The observations are a random sample from that population.

CLRM3 — variation in \(x\). The \(x_i\) are not all equal.

CLRM4 — zero conditional mean. \(E(u \mid x) = 0\).

Nothing about the spread of the errors is assumed yet, and nothing about their shape.

Textbooks label these differently. Wooldridge calls them SLR1 to SLR6 in the simple regression case and MLR1 to MLR6 with several explanatory variables; Gujarati and others speak of the classical linear regression model. The assumptions are the same and only the labels move.

Write the slope in deviation form and substitute the model. With \(\text{SST}_x = \sum(x_i - \bar{x})^2\),

\[ \begin{aligned} \hat{\beta}_1 &= \frac{\sum(x_i - \bar{x})y_i}{\text{SST}_x} && \text{since } \textstyle\sum(x_i-\bar{x})\bar{y} = 0 \\[4pt] &= \frac{\sum(x_i - \bar{x})(\beta_0 + \beta_1 x_i + u_i)}{\text{SST}_x} && \text{substituting CLRM1} \\[4pt] &= \beta_1 + \frac{\sum(x_i - \bar{x})u_i}{\text{SST}_x} \end{aligned} \]

because \(\sum(x_i - \bar{x}) = 0\) kills the \(\beta_0\) term and \(\sum(x_i - \bar{x})x_i = \text{SST}_x\) leaves \(\beta_1\).

That last line deserves to be pulled out and looked at on its own.

\[\hat{\beta}_1 \;=\; \underbrace{\beta_1}_{\text{truth}} \;+\; \underbrace{\frac{\sum(x_i - \bar{x})u_i}{\text{SST}_x}}_{\text{sampling error}}\]

Estimate = truth + sampling error.

The equation separates two quite different things.

The first term, \(\beta_1\), is the quantity we are trying to estimate. It is a fixed feature of the population and it is identical in every sample.

The second term is the price of observing only one sample. It depends entirely on the unobserved errors \(u_i\) — on which particular students happened to be drawn and how far each of them sat from the line. It is different in every sample, and it is the whole reason the grey lines in Figure 7.3 fan out.

So the question of whether OLS is centred correctly comes down to one thing: does that second term average out to zero?

Taking expectations conditional on the \(x\)’s, and using CLRM4 to set \(E(u_i \mid x) = 0\),

\[E(\hat{\beta}_1) = \beta_1 + \frac{\sum(x_i - \bar{x})E(u_i \mid x)}{\text{SST}_x} = \beta_1\]

It does. Across repeated samples \(\hat{\beta}_1\) is centred on \(\beta_1\), which is the first question mark in Table 7.1 filled in, and what the histogram was showing.

The proof is worth having for a second reason. It shows that unbiasedness hangs on exactly one place — the second term — and on exactly one condition, that the average error does not change with \(x\).

If students who attend more classes also happen to be more motivated, then the errors are systematically larger at larger \(x\). The second term no longer vanishes, and the estimator is biased by precisely its expectation.

That single term is why omitted variables matter, and Section 7.8 returns to it with this equation in hand.