8.12 Mathematical note: multiple regression in matrix form
Nothing in this unit depends on what follows. It is here for readers who would like to see the general case written compactly, and it can be skipped without consequence.
The estimator. With \(k\) explanatory variables, writing out \(k+1\) normal equations and solving them by hand is unpleasant. Matrix notation does the same work in three lines.
Collect the observations into an \(n \times 1\) vector \(Y\) and an \(n \times (k+1)\) matrix \(X\) whose first column is all ones and whose remaining columns are the explanatory variables. The model is
\[Y = X\beta + u\]
The sum of squared residuals is \((Y - X\beta)'(Y - X\beta)\). Differentiating with respect to \(\beta\) and setting the result to zero,
\[-2X'Y + 2X'X\beta = 0 \qquad\Longrightarrow\qquad X'X\hat{\beta} = X'Y\]
which is the whole system of normal equations in one line. Provided \(X'X\) can be inverted,
\[\hat{\beta} = (X'X)^{-1}X'Y\]
The variance. Substituting \(Y = X\beta + u\) gives \(\hat{\beta} = \beta + (X'X)^{-1}X'u\). The second term has expectation zero under CLRM4, which is unbiasedness. Taking the variance and using \(\mathrm{Var}(Au) = A\,\mathrm{Var}(u)\,A'\), then imposing homoskedasticity,
\[\mathrm{Var}(\hat{\beta}) = \sigma^2 (X'X)^{-1}\]
This is a \((k+1) \times (k+1)\) matrix. Its diagonal holds the variances of the individual coefficients — the square roots of which are the standard errors R prints — and its off-diagonal entries hold the covariances between them, which is what the \(F\) test of Section 8.6 needs in order to test several coefficients jointly.
Checking it. The formula is short enough to compute directly.
X <- cbind(1, students$attendance, students$cum_gpa, students$admission_score)
y <- students$sem_gpa
b_hat <- solve(t(X) %*% X) %*% t(X) %*% y
round(drop(b_hat), 5)#> [1] -7.6603 0.1360 0.5748 0.0835The same four numbers lm() reported in Section 8.2.
Where collinearity appears. The requirement in Section
8.4 that no explanatory variable be an exact linear
combination of the others is exactly the requirement that \(X'X\) be invertible.
If one column of \(X\) can be written in terms of the others, \(X\) is not of full
rank, \(X'X\) is singular, and there is no \(\hat{\beta}\) to compute — which is
what R is reporting when it returns NA.