Summary
- Two groups produce two sample means, and a difference between them is
guaranteed to appear even when the populations are identical. The question is
whether it is larger than sampling variation would produce.
- \(E[\bar{X} - \bar{Y}] = \mu_x - \mu_y\): the difference of sample means is
unbiased for the difference of population means.
- The variances add even though the means subtract, so
\(\mathrm{se}(\bar{X} - \bar{Y}) = \sqrt{\sigma_x^2/n + \sigma_y^2/m}\).
Comparing two groups is noisier than measuring one, and needs more data.
- With \(\sigma\) known the test is a \(Z\)-test on the difference; the interval is
\((\bar{x} - \bar{y}) \pm Z_{\alpha/2}\,\mathrm{se}\), and it excludes zero
exactly when the test rejects.
- With \(\sigma\) unknown, three cases differ only under the square root: large
samples use \(s_x\) and \(s_y\) with the normal; pooling assumes equal
variances and buys \(n + m - 2\) degrees of freedom; Welch assumes nothing
and adjusts the degrees of freedom down. R’s
t.test() uses Welch unless you
pass var.equal = TRUE, and Welch is the safer default.
- A paired design gives one number per unit. Take differences and run a
one-sample test. Treating paired data as two independent samples discards the
comparison the design was built to make — in the marriage data it turned
\(p = 0.015\) into \(p = 0.61\).
- A significant difference need not be an important one. Read the interval in
the units of the problem.
- A difference between two groups that formed themselves is not an effect. The
test compares the groups as they arrive and is silent about why they differ.