9.8 R-squared always rises

Can we use \(R^2\) to decide whether a variable belongs?

The suggestion is natural. \(R^2\) measures how much of the variation the model accounts for, so a variable that improves the model should raise it.

The difficulty is that every variable raises it.

students <- read.csv("data/attendance-grades.csv")

set.seed(2)
noise <- as.data.frame(matrix(rnorm(nrow(students) * 10), ncol = 10))
augmented <- cbind(students, noise)

base <- lm(sem_gpa ~ attendance + cum_gpa + admission_score, data = students)
junk <- lm(reformulate(c("attendance", "cum_gpa", "admission_score",
                         names(noise)), "sem_gpa"), data = augmented)

round(c(r2_base        = summary(base)$r.squared,
        r2_plus_junk   = summary(junk)$r.squared,
        adj_base       = summary(base)$adj.r.squared,
        adj_plus_junk  = summary(junk)$adj.r.squared), 4)

#>       r2_base  r2_plus_junk      adj_base adj_plus_junk 
#>        0.5421        0.5464        0.5401        0.5375

The ten added variables are pure random noise, generated without reference to the data. They cannot possibly explain semester GPA.

\(R^2\) rises anyway, from 0.5421 to 0.5464.

\(R^2\) can never fall when a variable is added.

Least squares could always set the new coefficient to zero and reproduce the old fit exactly. Since it minimises the sum of squared residuals, it will only depart from zero if doing so fits this sample better — which pure noise occasionally does, by chance.

A measure that never falls cannot be used to decide whether something belongs.

This is true even if the new variable is generated by rolling dice.

Adjusted R-squared

Adjusted \(R^2\) penalises a model for estimating more parameters. A variable must improve the fit by more than the price of the parameter it costs, or the measure goes down.

In the output above, adjusted \(R^2\) falls from 0.5401 to 0.5375. It has detected what \(R^2\) could not.

The adjustment works by dividing each sum of squares by its degrees of freedom before comparing them:

\[\bar{R}^2 = 1 - \frac{\text{SSR}/(n-k-1)}{\text{SST}/(n-1)} \qquad\text{equivalently}\qquad \bar{R}^2 = 1 - \frac{(1-R^2)(n-1)}{n-k-1}\]

Adding a variable raises \(k\) and so shrinks \(n-k-1\), which inflates \(\text{SSR}/(n-k-1)\) unless SSR falls by enough to compensate.

The formula is worth seeing once. It is almost never worth computing by hand, since every regression command reports it.

Adjusted \(R^2\) can be negative, and it is not a proportion of anything. It is a model comparison device, not a measure of fit.

It is also a weak one. It penalises parameters, not irrelevance, and a variable that is genuinely useless but happens to correlate with the outcome in this sample will still raise it.