5.4 When σ is unknown
As always, we do not know \(\sigma_x\) or \(\sigma_y\). Now there are two of them.
In practice both population standard deviations must be estimated from the samples, using \(s_x\) and \(s_y\). As in Section 4.7, that costs something, and what it costs depends on the situation. Three cases arise, and they differ only in what goes underneath the square root.
Large samples
If both samples are reasonably large — a common rule of thumb is at least 20 each — then \(s_x\) and \(s_y\) are reliable enough that substituting them changes little, and the normal distribution remains a good approximation:
\[Z_{calc} = \frac{\bar{x} - \bar{y}}{\sqrt{\dfrac{s_x^2}{n} + \dfrac{s_y^2}{m}}}\]
This is the same statistic with hats on the standard deviations.
Equal variances: the pooled estimator
If we are willing to assume the two populations share a common standard deviation, \(\sigma_x = \sigma_y = \sigma\), then both samples are estimating the same quantity and it is wasteful to estimate it twice. We combine them.
The pooled variance estimator is a weighted average of the two sample variances, each weighted by its degrees of freedom:
\[s_p^2 = \frac{(n-1)s_x^2 + (m-1)s_y^2}{n + m - 2}\]
and the test statistic becomes
\[t_{calc} = \frac{\bar{x} - \bar{y}}{\sqrt{s_p^2\left(\dfrac{1}{n} + \dfrac{1}{m}\right)}}\]
compared against \(t\) with \(n + m - 2\) degrees of freedom.
The degrees of freedom are worth pausing on. We began with \(n + m\) observations, and spent one estimating each of the two sample means, leaving \(n + m - 2\). The same accounting as Section 3.2, applied twice.
Pooling buys precision — \(n + m - 2\) degrees of freedom rather than something closer to the smaller sample’s — but only if the assumption holds. If one group really is more variable than the other, the pooled estimate is a weighted average of two different things, and the test is wrong in a way the output will not reveal.
Unequal variances: Welch’s test
The safer option makes no assumption about the variances. It uses the separate estimates,
\[t_{calc} = \frac{\bar{x} - \bar{y}}{\sqrt{\dfrac{s_x^2}{n} + \dfrac{s_y^2}{m}}}\]
and adjusts the degrees of freedom downwards to compensate for the extra uncertainty. The adjustment is not a whole number and is not worth memorising; R computes it.
R’s t.test() uses Welch by default. If you have derived a pooled
statistic by hand and R disagrees with you, this is usually why.
To pool, pass var.equal = TRUE explicitly.
Modern practice is to prefer Welch. It costs very little when the variances really are equal, and it protects you when they are not — and you can never verify equal variances, only fail to reject the claim that they are equal.
Do North and South Indian students differ in height?
student-survey-height.csv records 100 students, each classified by region.
heights <- read.csv("data/student-survey-height.csv")
aggregate(height ~ region, data = heights,
function(x) round(c(n = length(x), mean = mean(x), sd = sd(x)), 2))#> region height.n height.mean height.sd
#> 1 North Indian 57.00 165.16 7.29
#> 2 South Indian 43.00 163.67 9.85The two sample means differ by about 1.5 cm, and the two sample standard deviations are not equal — 7.3 against 9.9. That is a case for Welch.
#>
#> Welch Two Sample t-test
#>
#> data: height by region
#> t = 0.83, df = 74, p-value = 0.4
#> alternative hypothesis: true difference in means between group North Indian and group South Indian is not equal to 0
#> 95 percent confidence interval:
#> -2.074 5.043
#> sample estimates:
#> mean in group North Indian mean in group South Indian
#> 165.2 163.7\(t = 0.83\) on 74.4 degrees of freedom, with \(p = 0.41\). We fail to reject. The 95% interval for the difference runs from \(-2.07\) to \(5.04\) cm, and it contains zero — the two statements agree, as they must.
So a 1.5 cm gap in the samples is entirely unremarkable. With these sample sizes and this much variation within each group, differences of that size turn up constantly when the populations are identical.
Compare the degrees of freedom the two approaches give here.
c(welch = t.test(height ~ region, data = heights)$parameter,
pooled = t.test(height ~ region, data = heights, var.equal = TRUE)$parameter)#> welch.df pooled.df
#> 74.36 98.00Pooling claims 98 degrees of freedom; Welch allows only 74.4. The difference is the price of not assuming the two groups are equally variable — and here, where one sample’s standard deviation is a third larger than the other’s, it is a price worth paying.
Both give the same verdict in this case. They do not always.