8.4 The assumptions of multiple regression
One explanatory variable has become several. Does that require a completely new set of assumptions?
Surprisingly, very little changes.
Everything that made OLS work in Unit 7 still matters. We still need a model that is linear in the parameters, a random sample, a conditional mean that is correctly specified, and — if we want the classical standard errors — constant error variance.
The only genuinely new assumption is one that could not arise before: the explanatory variables cannot perfectly explain one another.
CLRM1 — linear in parameters. \(y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u\).
CLRM2 — random sampling. The \(n\) observations are a random sample from that population.
CLRM3 — variation in the explanatory variables. No \(x_j\) takes the same value for every observation.
CLRM4 — zero conditional mean. \(E(u \mid x_1, \ldots, x_k) = 0\).
CLRM5 — homoskedasticity. \(\mathrm{Var}(u \mid x_1, \ldots, x_k) = \sigma^2\).
CLRM6 — normality. The errors are normally distributed. As before, this is needed only for exact small-sample inference.
CLRM7 — no perfect collinearity. No explanatory variable is an exact linear combination of the others.
Under CLRM1–CLRM4, OLS remains unbiased, exactly as before. The proof is identical to Section 7.4, written in matrix notation rather than scalar notation; Section 8.12 gives the compact version.
Three assumptions deserve comment.
Zero conditional mean
The assumption has not changed. The error term must still be unrelated to the explanatory variables,
\[E(u \mid x_1, \ldots, x_k) = 0\]
What has changed is what remains inside \(u\).
In the simple regression, prior GPA and admission score were omitted variables. Because they were correlated with attendance, they threatened the assumption directly.
In the multiple regression they are no longer omitted. They have become explanatory variables.
That is the sense in which multiple regression improves on simple regression. It does not make the assumption true. It removes some of the reasons why it might fail.
Whatever is still omitted — motivation, ability, family background — remains inside \(u\), and can still bias the coefficients if it is related to the explanatory variables.
No perfect collinearity
This assumption is not about economics at all. It is a requirement for the regression to be mathematically identifiable.
Our data contain a violation, and it is instructive.
#> [1] 32 32 32 32 32 32The course had 32 classes, so attendance and classes missed always sum to 32. Knowing one is knowing the other. Ask R to use both.
#> (Intercept) attendance classes_missed
#> 1.562 0.189 NAR returns NA for the second coefficient. It has not failed; it is reporting
that the question has no answer.
Why there is no answer
The reason can be seen with one substitution. Since \(\text{missed} = 32 - \text{attendance}\), the fitted part of the model is
\[ \begin{aligned} \beta_0 + \beta_1\,\text{att} + \beta_2\,\text{missed} &= \beta_0 + \beta_1\,\text{att} + \beta_2(32 - \text{att}) \\[4pt] &= \underbrace{(\beta_0 + 32\beta_2)}_{\text{intercept}} + \underbrace{(\beta_1 - \beta_2)}_{\text{slope}}\,\text{att} \end{aligned} \]
The two coefficients enter only through their difference. Any pair with the same \(\beta_1 - \beta_2\) produces exactly the same fitted values, and therefore exactly the same sum of squared residuals.
Least squares chooses the coefficients that minimise that sum. Here infinitely many pairs minimise it equally well, so there is nothing to choose between them.
att <- students$attendance
mis <- students$classes_missed
## Three quite different pairs, all with the same difference b1 - b2.
fit_at <- function(b0, b1, b2) b0 + b1 * att + b2 * mis
round(c(pair1_vs_pair2 = max(abs(fit_at(1.562, 0.189, 0) -
fit_at(1.562 - 32, 1.189, 1))),
pair1_vs_pair3 = max(abs(fit_at(1.562, 0.189, 0) -
fit_at(1.562 - 160, 5.189, 5)))), 10)#> pair1_vs_pair2 pair1_vs_pair3
#> 0 0A slope of 0.189 on attendance with 0 on classes missed, a slope of 1.189 with 1, and a slope of 5.189 with 5 all fit the data identically. The differences are zero to machine precision.
The data cannot distinguish between them, and neither can any estimator.
The same fact appears in the two-step procedure of Section 8.3.
Regress attendance on classes missed, and the fit is perfect: \(R^2 = 1\), and every residual is zero. Step two would then regress GPA on a variable that does not vary at all.
That is assumption CLRM3 failing, and it is also the variance formula of the next subsection with \(R_j^2 = 1\), so that \((1 - R_j^2) = 0\) and the denominator vanishes. Perfect collinearity is not a separate problem from the one the next subsection describes. It is the same problem at its limit.
The coefficient on attendance is supposed to answer: what happens to the grade when attendance rises by one, holding classes missed fixed?
Nothing happens, because nothing can happen. Attending one more class means missing one fewer. There is no student for whom one changed and the other did not, so the data contain no information about that comparison, and no amount of cleverness can extract it.
When controls tell us the same thing
Perfect collinearity is rare. The important case is when two explanatory variables contain much of the same information.
Attendance and prior GPA are not identical. But students with high prior GPAs also tend to attend more classes. Part of the variation in attendance can therefore already be predicted from prior GPA.
That is exactly what Section 8.3 removed during the residualisation step.
The more predictable attendance becomes, the less independent variation remains for estimating its coefficient. Less variation means less information. Less information means larger standard errors.
The variance formula puts a number on it:
\[\mathrm{Var}(\hat{\beta}_j) = \frac{\sigma^2}{\sum_{i=1}^{n}(x_{ij} - \bar{x}_j)^2 \, (1 - R_j^2)}\]
Compare it with the simple regression formula of Section 7.5: everything is the same except the extra factor \((1-R_j^2)\) in the denominator.
\(R_j^2\) is not the \(R^2\) from the regression you care about.
It is the \(R^2\) from regressing one explanatory variable on all the others. It tells us how much of that variable is already predictable before the regression even begins.
If \(R_j^2 = 0\), the variable contributes completely new information.
If \(R_j^2 = 0.95\), almost everything it contains was already available from the remaining explanatory variables. OLS is then trying to estimate a coefficient using only the tiny amount of variation left over.
That is why the standard error grows.
About 29% of attendance can already be predicted from prior GPA and admission score. The coefficient therefore relies on the remaining 71% — the unexpected part of attendance. There is still plenty of independent variation left, so the increase in the standard error is modest.
As before, estimating additional coefficients consumes degrees of freedom, so the residual variance is now divided by \(n-k-1\).
The trade
Multiple regression always involves a trade-off.
Every additional control makes the comparison more credible by removing another potential source of bias.
But every additional control also consumes variation, making the coefficient less precise.
Good empirical work is largely about deciding which controls are worth that price.