Summary
- The multiple regression model is
\(y = \beta_0 + \beta_1x_1 + \cdots + \beta_kx_k + u\). Taking conditional
expectations, \(E[y \mid x_1,\ldots,x_k]\) is the population regression function,
now a plane rather than a line.
- Each \(\beta_j\) is a partial derivative: the change in average \(y\) for a
one-unit rise in \(x_j\), holding the other explanatory variables fixed. This is
the ceteris paribus interpretation, and it is why multiple regression exists.
- “Holding fixed” is an operation on the data, not an experiment. By the
Frisch–Waugh–Lovell result, \(\hat{\beta}_j\) can be obtained by regressing
\(x_j\) on the other regressors, keeping the residuals, and regressing \(y\) on
those residuals alone.
- It follows that the coefficient uses only the variation in \(x_j\) that the other
variables cannot explain. Controls remove variation, and removing variation
costs precision.
- The assumptions are those of Unit 7 with the conditioning extended to
all regressors, plus CLRM7, no perfect collinearity. A violation is not a
statistical problem but an arithmetic one: the comparison the coefficient
refers to does not exist in the data, and R returns
NA.
- \(\mathrm{Var}(\hat{\beta}_j) = \sigma^2 / [\sum(x_{ij}-\bar{x}_j)^2(1-R_j^2)]\),
where \(R_j^2\) comes from regressing \(x_j\) on the other regressors. The larger
\(R_j^2\), the more the control costs. Also
\(\hat{\sigma}^2 = \sum \hat{u}_i^2/(n-k-1)\).
- Testing a single coefficient is unchanged:
\(t = \hat{\beta}_j/\mathrm{se}(\hat{\beta}_j)\) on \(n-k-1\) degrees of freedom.
- Testing several at once requires the F test, which compares a restricted
model against an unrestricted one:
\(F = [(\text{SSR}_r - \text{SSR}_{ur})/q] \big/ [\text{SSR}_{ur}/(n-k-1)]\). It
is always a right-tailed test.
- Separate \(t\)-tests are not a substitute. Correlated regressors can be
individually insignificant and jointly overwhelming, because each \(t\)-test uses
only the part of its variable that the others cannot explain.
- \(F\) is the ratio of two independent variance estimates, each divided by its
degrees of freedom. Under the null both estimate \(\sigma^2\), so \(F\) averages
about one. Its distribution is right-skewed, so most of the mass lies below
that average.
- Omitted variable bias is \(\beta_2\delta_1\) — the effect of the omitted
variable on \(y\), times its relationship with the included one. It vanishes if
either factor is zero, and its sign is the product of their signs.
- Controlling for measured variables removes their contribution to the bias and
nothing else. Multiple regression narrows the gap to a causal effect without
closing it, and reports nothing about what remains.