5.8 Practice problems
Work each one before opening the solution. The question bank that follows has no answers.
Concept checks
5.1 Why is the standard error of a difference of means larger than the standard error of either mean on its own?
Because the two sample means each carry sampling variation, and both end up in the difference. Formally, \(\mathrm{Var}(aX) = a^2\mathrm{Var}(X)\) with \(a = -1\) gives \((-1)^2 = +1\), so the variances add:
\[\mathrm{Var}[\bar{X} - \bar{Y}] = \frac{\sigma_x^2}{n} + \frac{\sigma_y^2}{m}\]
Subtracting a noisy quantity does not cancel noise; it adds more. This is why detecting a difference between two groups takes more data than estimating one group’s mean to the same precision.
5.2 A study reports that two groups differ with \(p = 0.003\). A colleague concludes the difference is large. What is wrong?
The \(p\)-value measures how incompatible the data are with the claim of no difference. It says nothing about the size of the difference.
A very small \(p\)-value can come from a tiny difference measured in a large sample. The size of the difference is reported by the confidence interval, in the units of the problem, and that is what has to be inspected before calling a result large or important.
5.3 Why does R’s t.test() report fractional degrees of freedom?
Because it performs Welch’s test by default, which does not assume the two populations have equal variances. The correction for that extra uncertainty adjusts the degrees of freedom downwards, and the adjustment is generally not a whole number.
The pooled test, obtained with var.equal = TRUE, uses \(n + m - 2\) degrees of
freedom, which always is.
5.4 The same twenty households are surveyed before and after a cash transfer. Why is a two-sample test the wrong tool?
Because the two columns are not independent samples — each household appears in both. The design produces one number per household, the change.
Households differ enormously among themselves, and that variation is present identically in both columns. A two-sample test treats it as noise, and it swamps the change we are trying to detect. Taking differences within households removes it, leaving only the quantity of interest.
5.5 Villages that received a road show higher incomes than villages that did not. State two distinct reasons this need not mean the road raised incomes.
First, the villages may have differed beforehand. Roads are not built at random — they tend to reach places that are more accessible, more populous, more politically connected or already better off. Those same features raise incomes on their own.
Second, the two groups may differ in things that changed alongside the road: electrification, a market, a school. The road is correlated with a bundle of improvements, and a comparison of means attributes the whole bundle to it.
In both cases the arithmetic is correct and the causal reading is not.
Explain
5.6 Explain the trade-off between the pooled and Welch tests, and why modern practice prefers Welch.
Pooling assumes the two populations share a standard deviation. If true, both samples estimate the same quantity, they can be combined, and the resulting test has \(n + m - 2\) degrees of freedom — more than Welch allows, so more power.
If false, the pooled estimate averages two different quantities and the test is wrong in a way nothing in the output reveals. Welch avoids the assumption entirely, at the cost of fewer degrees of freedom.
The reason to prefer Welch is asymmetric risk. When the variances really are equal, Welch loses very little. When they are not, pooling misleads. And the assumption cannot be verified — a test of equal variances can only fail to reject, which is not the same as establishing it.
5.7 A researcher tests whether men and women in a firm are paid differently, finds \(p = 0.02\), and concludes the firm discriminates. Evaluate.
The test establishes that average pay differs by more than sampling variation would explain. That is not the same as discrimination.
Men and women in the firm may differ in role, seniority, years of experience, hours, or field of training, and any of those could account for a pay gap without anyone being treated differently for the same work. The comparison of raw means attributes the whole difference to sex because sex is the only variable in the analysis.
Establishing discrimination requires comparing like with like — people doing the same work with the same experience — which is a question of design and, in later units, of holding other variables constant. The two-sample test is not capable of it.
Numerical problems
5.8 Two irrigation methods are trialled. Drip irrigation on 15 plots yields a mean of 38.4 quintals per hectare; flood irrigation on 18 plots yields 35.9. Population standard deviations are known to be 3.2 and 4.1 respectively.
- Test at the 5% level whether the mean yields differ.
- Construct a 95% confidence interval for the difference.
- Does the interval agree with the test?
(i) The standard error of the difference is
\[\sqrt{\frac{3.2^2}{15} + \frac{4.1^2}{18}} = \sqrt{0.6827 + 0.9339} = \sqrt{1.6166} = 1.271\]
\[Z_{calc} = \frac{38.4 - 35.9}{1.271} = \frac{2.5}{1.271} = 1.967\]
Since \(1.967 > 1.96\), we reject \(H_0\) at the 5% level — but only just.
(ii) \(2.5 \pm 1.96(1.271) = 2.5 \pm 2.491 = [0.009,\ 4.991]\).
(iii) Yes. The interval excludes zero, and the test rejects. Both do so by the narrowest possible margin, which is the same fact stated twice: the interval endpoint sits at 0.009 because \(Z_{calc}\) sits at 1.967.
A result this marginal should be reported as marginal. Nothing about it justifies confidence that drip irrigation is better.
5.9 Two samples give \(n = 10\), \(s_x = 4\) and \(m = 14\), \(s_y = 6\).
- Compute the pooled variance estimate.
- How many degrees of freedom does the pooled test use?
- Why is it not \(n + m\)?
(i)
\[s_p^2 = \frac{(10-1)16 + (14-1)36}{10 + 14 - 2} = \frac{144 + 468}{22} = \frac{612}{22} = 27.82\]
so \(s_p = 5.27\), lying between 4 and 6 and closer to 6 because the larger sample carries more weight.
(ii) \(n + m - 2 = 22\).
(iii) Because two quantities were estimated from the data before the spread could be measured — one sample mean in each group. Each costs a degree of freedom, exactly as in Section 3.2, where estimating a single mean cost one.
5.10 A programme is evaluated by comparing 200 participants with 200 non-participants. Participants’ mean income is ₹4,200 higher, with a standard error of ₹1,400 for the difference.
- Test whether the difference is zero at the 5% level.
- Construct a 95% interval.
- The programme was open to anyone who applied. What can be concluded about its effect?
(i) \(Z_{calc} = 4200/1400 = 3.0\), which exceeds 1.96. We reject the claim that the two groups have the same mean income.
(ii) \(4200 \pm 1.96(1400) = [₹1{,}456,\ ₹6{,}944]\).
(iii) Nothing about its effect. Because the programme was open to applicants, participants selected themselves. People who apply to income programmes may be more motivated, better informed, or better connected — or, equally plausibly, more desperate and worse off to begin with. The direction of the bias is not even clear.
The interval is a correct statement about the difference between two groups of people. It is not an estimate of what the programme did to anybody.
R exercises
5.11 Using data/student-survey-sleep.csv, test whether urban and rural
students report different average hours of sleep. Report the interval and say
what you conclude.
sleep <- read.csv("data/student-survey-sleep.csv")
aggregate(sleep ~ residence, data = sleep,
function(x) round(c(n = length(x), mean = mean(x), sd = sd(x)), 2))#> residence sleep.n sleep.mean sleep.sd
#> 1 Rural 21.00 6.52 1.53
#> 2 Urban 85.00 6.60 1.12#>
#> Welch Two Sample t-test
#>
#> data: sleep by residence
#> t = -0.21, df = 26, p-value = 0.8
#> alternative hypothesis: true difference in means between group Rural and group Urban is not equal to 0
#> 95 percent confidence interval:
#> -0.8064 0.6540
#> sample estimates:
#> mean in group Rural mean in group Urban
#> 6.524 6.600The two group means are close and the interval for the difference contains zero, so we fail to reject. There is no evidence here that urban and rural students sleep different amounts on average.
Note what “no evidence of a difference” is not: evidence of no difference. With these sample sizes, a real difference of well under an hour would be difficult to detect at all, and the interval says as much by being wide.
5.12 Test whether social media use differs between North and South Indian students in the same dataset, using both Welch and the pooled test. Do they agree, and which would you report?
rbind(
welch = unlist(t.test(socialmediahours ~ region, data = sleep)[c("statistic", "parameter", "p.value")]),
pooled = unlist(t.test(socialmediahours ~ region, data = sleep, var.equal = TRUE)[c("statistic", "parameter", "p.value")])
)#> statistic.t parameter.df p.value
#> welch 0.9172 50.63 0.3634
#> pooled 0.9845 104.00 0.3271The two agree closely here, which is what happens when the sample standard deviations are similar — the pooled and separate estimates of the variance are then nearly the same number.
Report Welch. It gives the same answer when the assumption holds and a correct one when it does not, so there is no situation in which pooling is the safer choice.
5.13 Take the wage data as a population and draw independent samples of 200 urban and 200 rural workers. Test whether mean annual earnings differ, and report the interval. Then say what the interval does and does not establish.
wages <- read.csv("data/wages-india-synthetic.csv")
set.seed(9)
urban <- sample(wages$annual_earnings[wages$residence == "Urban"], 200)
rural <- sample(wages$annual_earnings[wages$residence == "Rural"], 200)
t.test(urban, rural)#>
#> Welch Two Sample t-test
#>
#> data: urban and rural
#> t = 6.5, df = 223, p-value = 0.0000000005
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#> 47095 88103
#> sample estimates:
#> mean of x mean of y
#> 100811 33212The difference is large and the interval is nowhere near zero, so urban workers in this population earn substantially more than rural workers.
What the interval establishes is the size of that gap. What it does not establish is that moving a worker from a village to a city would raise their earnings by that amount. Urban and rural workers differ in education, occupation and industry, and the sample says nothing about which part of the gap belongs to the place and which to the people.
5.14 Simulate the point that variances add. Draw two independent samples of 100 from a population with \(\sigma = 10\), and compare the observed standard deviation of \(\bar{x}\) with that of \(\bar{x} - \bar{y}\) across 5,000 repetitions. Check both against theory.
set.seed(4)
sims <- replicate(5000, {
x <- rnorm(100, mean = 50, sd = 10)
y <- rnorm(100, mean = 50, sd = 10)
c(one_mean = mean(x), difference = mean(x) - mean(y))
})
round(c(sd_of_one_mean = sd(sims["one_mean", ]),
theory_one_mean = 10 / sqrt(100),
sd_of_difference = sd(sims["difference", ]),
theory_difference = sqrt(10^2/100 + 10^2/100)), 4)#> sd_of_one_mean theory_one_mean sd_of_difference theory_difference
#> 0.9989 1.0000 1.4260 1.4142The difference is about \(\sqrt{2}\) times as variable as a single mean, exactly as \(\sqrt{\sigma^2/n + \sigma^2/m}\) predicts. Note also that the difference is centred on zero: both populations have mean 50, and the estimator is unbiased for a difference of zero.
5.15 Demonstrate the cost of ignoring pairing. Using marriage-ages.csv,
report the paired and unpaired \(p\)-values, and explain the gap by comparing the
standard deviation of the differences with the standard deviations of the two
columns.
couples <- read.csv("data/marriage-ages.csv")
c(paired = t.test(couples$partner1, couples$partner2, paired = TRUE)$p.value,
unpaired = t.test(couples$partner1, couples$partner2)$p.value)#> paired unpaired
#> 0.01496 0.60699round(c(sd_partner1 = sd(couples$partner1),
sd_partner2 = sd(couples$partner2),
sd_of_diffs = sd(couples$partner1 - couples$partner2)), 3)#> sd_partner1 sd_partner2 sd_of_diffs
#> 11.156 10.174 2.635The two columns each have a standard deviation of about 11 years, because the couples range from their mid-thirties to their seventies. The differences have a standard deviation of about 2.6 years.
That is the whole explanation. The unpaired test measures the small gap against the huge between-couple variation and sees nothing. The paired test removes that variation — it is common to both columns and cancels on subtraction — and measures the gap against what is left.
Paired designs
5.16 A paired test on \(n\) pairs has \(n - 1\) degrees of freedom, not \(2n - 2\). Why, when there are \(2n\) numbers in the dataset?
Because the analysis is not performed on the \(2n\) numbers. It is performed on the \(n\) differences.
Taking differences reduces \(2n\) observations to \(n\), one per pair. From those \(n\) we estimate one quantity, \(\bar{d}\), before we can measure their spread, so one degree of freedom is spent and \(n - 1\) remain — exactly as in the one-sample case of Section 3.2.
The \(2n - 2\) of the pooled two-sample test is right for a design in which the \(2n\) observations really are \(2n\) independent pieces of information. In a paired design they are not: knowing one member of a pair tells you a great deal about the other.
5.17 What feature of the data determines whether pairing was worth doing?
The correlation between the two columns.
Pairing removes whatever is common to both members of a pair. If the two measurements are strongly related — the same household in two rounds, the same plot in two seasons — then a great deal is common, differencing removes it, and the standard error falls sharply.
If the two columns happen to be unrelated, there is nothing common to remove. Differencing then leaves the variability essentially unchanged, and the paired test performs about as well as the unpaired one, losing only the few degrees of freedom.
So pairing rarely costs much and can gain a great deal. What determines which happens is not a choice; it is a property of the design.
5.18 A researcher measures productivity at the same 30 firms before and after a policy change, then runs a two-sample \(t\)-test. Describe what will happen to the \(p\)-value and why.
It will be too large — often dramatically so — and the researcher will under-report a real effect.
Firms differ enormously among themselves in productivity: a large firm and a small one may differ by a factor of ten, while the policy might move any single firm by a few per cent. In a two-sample test that between-firm variation is counted as noise, and it swamps the change being looked for.
The pairing is exactly what removes it. Each firm’s before-measurement acts as its own comparison, so the firm’s general productivity level cancels and only its change remains. Writing \(x_i = \mu + \alpha_i + \delta_i\) and \(y_i = \mu + \alpha_i + \varepsilon_i\), the large firm effect \(\alpha_i\) appears in both and disappears on subtraction.
The error is conservative rather than anti-conservative, which is why it often goes unnoticed: the researcher simply concludes there is no effect.
5.19 A new irrigation technique is trialled on eight plots. Yields in quintals per hectare before and after are
| Plot | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Before | 32 | 28 | 35 | 30 | 33 | 29 | 31 | 34 |
| After | 35 | 33 | 37 | 36 | 37 | 30 | 36 | 40 |
- Explain why this is a paired design.
- Compute the differences, \(\bar{d}\) and \(s_d\).
- Test at the 5% level whether the technique changed yields.
- Construct a 95% confidence interval for the mean change.
(i) Each plot is measured twice. The two columns are not eight plots and eight other plots; they are the same eight plots under two conditions, so every observation in one column has a natural partner in the other. Plots differ in soil, slope and drainage, and those differences are present identically in both rounds.
(ii) Differences (after \(-\) before): \(3, 5, 2, 6, 4, 1, 5, 6\).
\[\bar{d} = \frac{32}{8} = 4.0\]
Deviations from 4: \(-1, 1, -2, 2, 0, -3, 1, 2\), whose squares sum to 24, so
\[s_d^2 = \frac{24}{7} = 3.4286 \qquad s_d = 1.852\]
and the standard error is \(1.852/\sqrt{8} = 0.655\).
(iii)
\[t = \frac{4.0}{0.655} = 6.11\]
with \(df = 7\). The critical value is \(t_{0.025,\,7} = 2.365\), and \(6.11 > 2.365\), so we reject \(H_0: \mu_d = 0\). The \(p\)-value is 0.0005.
(iv) \(4.0 \pm 2.365(0.655) = [2.45,\ 5.55]\) quintals per hectare.
Note the interval is in the units of the problem and excludes zero, agreeing with the test. Note also that the estimated gain of 4 quintals per hectare is large in agronomic terms as well as statistically detectable — the two questions are separate, and here both answers happen to be favourable.
5.20 Seven workers are timed on a task, in minutes, before and after training.
| Worker | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|
| Before | 14 | 12 | 15 | 13 | 16 | 11 | 14 |
| After | 11 | 10 | 13 | 12 | 13 | 11 | 12 |
- State the hypotheses for a test that training reduced completion time.
- Compute the test statistic and its degrees of freedom.
- Test at the 5% level.
- Construct the appropriate one-sided 95% bound and interpret it.
(i) Working with \(d_i = \text{before}_i - \text{after}_i\), so that a reduction in time gives a positive difference:
\[H_0 : \mu_d \leq 0 \qquad H_A : \mu_d > 0\]
The equality lives in \(H_0\), as always, and the claim we want evidence for sits in \(H_A\).
(ii) Differences: \(3, 2, 2, 1, 3, 0, 2\). So \(\bar{d} = 13/7 = 1.857\) and \(s_d = 1.069\), giving a standard error of \(1.069/\sqrt{7} = 0.404\) and
\[t = \frac{1.857}{0.404} = 4.60\]
with \(df = 6\).
(iii) Right-tailed at 5%, so the critical value is \(t_{0.05,\,6} = 1.943\) — not 2.447. Since \(4.60 > 1.943\) we reject \(H_0\); the one-sided \(p\)-value is 0.0019.
(iv) A one-sided lower bound spends all of \(\alpha\) in one tail:
\[\bar{d} - t_{0.05,\,6}\,\frac{s_d}{\sqrt{n}} = 1.857 - 1.943(0.404) = 1.07\]
so the interval is \([1.07,\ \infty)\). With 95% confidence, training reduced completion time by at least 1.07 minutes. We have deliberately given up any statement about how large the reduction might be, because the question only asked whether there was one.
5.21 Using marriage-ages.csv, verify by direct computation that
paired = TRUE does nothing more than take differences. Then run the one-sided
test of whether partner 1 is older.
couples <- read.csv("data/marriage-ages.csv")
d <- couples$partner1 - couples$partner2
# Three routes to the same number
c(by_hand = mean(d) / (sd(d) / sqrt(length(d))),
one_sample = t.test(d, mu = 0)$statistic,
paired = t.test(couples$partner1, couples$partner2, paired = TRUE)$statistic)#> by_hand one_sample.t paired.t
#> 3 3 3Identical, as they must be: paired = TRUE subtracts the columns and calls the
one-sample routine.
#>
#> Paired t-test
#>
#> data: couples$partner1 and couples$partner2
#> t = 3, df = 9, p-value = 0.007
#> alternative hypothesis: true mean difference is greater than 0
#> 95 percent confidence interval:
#> 0.9724 Inf
#> sample estimates:
#> mean difference
#> 2.5The one-sided \(p\)-value is half the two-sided one, 0.0075 against 0.015, and the interval is now a lower bound: partner 1 is older by at least 1.02 years.
The one-sided test is defensible here only because the direction was decided before looking — we asked whether partner 1 is older, not whether the two differ. Choosing the direction after seeing that eight of ten differences were positive would not be a test at all.
5.22 Show that the benefit of pairing depends on the correlation between the two columns. Simulate pairs of measurements with correlation 0, 0.5 and 0.9, a true difference of 4, and \(n = 10\), and compare how often each test rejects.
set.seed(7)
power_at <- function(rho, reps = 4000) {
out <- replicate(reps, {
a <- rnorm(10)
b <- rho * a + sqrt(1 - rho^2) * rnorm(10)
x <- 50 + 10 * a
y <- 50 + 10 * b + 4 # true difference is 4
c(paired = t.test(x, y, paired = TRUE)$p.value < 0.05,
unpaired = t.test(x, y)$p.value < 0.05,
se_paired = sd(x - y) / sqrt(10),
se_unpaired = sqrt(var(x)/10 + var(y)/10))
})
c(rho = rho, rowMeans(out))
}
round(t(sapply(c(0, 0.5, 0.9), power_at)), 3)#> rho paired unpaired se_paired se_unpaired
#> [1,] 0.0 0.125 0.126 4.348 4.408
#> [2,] 0.5 0.212 0.073 3.069 4.402
#> [3,] 0.9 0.717 0.014 1.374 4.389Read the two standard error columns first, because they explain everything else. The paired standard error falls from about 4.35 to 1.37 as the correlation rises. The unpaired standard error sits near 4.40 throughout — it is computed from the two columns separately and so cannot notice that they are related at all.
The rejection rates follow. At \(\rho = 0\) the two tests are equivalent, and the paired one is marginally worse because it has 9 degrees of freedom against 18. At \(\rho = 0.9\) the paired test rejects about 72% of the time and the unpaired test almost never does.
That last figure deserves a moment. The unpaired test does not merely fail to improve as the correlation rises; it gets worse. With strongly correlated columns the observed difference is tightly concentrated near its true value of 4, while the standard error the unpaired test uses stays at 4.4 — so its statistic sits reliably near \(4/4.4 \approx 0.9\) and almost never strays past the critical value. Ignoring a strong pairing produces a test that is consistently, confidently unable to see anything.