5.6 A difference is not an effect
The test says two groups differ. Does it say why?
We can now establish that two groups differ by more than sampling variation. That is a genuine achievement, and it is not what most interesting questions actually ask.
Consider the attendance data from this course. Split the 680 students at the median attendance and compare their final examination scores.
students <- read.csv("data/attendance-grades.csv")
students$group <- ifelse(students$attendance >= median(students$attendance),
"High attendance", "Low attendance")
aggregate(final_score ~ group, data = students,
function(x) round(c(n = length(x), mean = mean(x)), 2))#> group final_score.n final_score.mean
#> 1 High attendance 346.0 91.3
#> 2 Low attendance 334.0 90.4#>
#> Welch Two Sample t-test
#>
#> data: final_score by group
#> t = 2.5, df = 671, p-value = 0.01
#> alternative hypothesis: true difference in means between group High attendance and group Low attendance is not equal to 0
#> 95 percent confidence interval:
#> 0.2063 1.5927
#> sample estimates:
#> mean in group High attendance mean in group Low attendance
#> 91.3 90.4Students who attended more scored higher, and \(p = 0.011\) rejects the claim that the two groups have the same mean.
It is tempting to conclude that attending class raises examination scores by about a point. Two separate problems stand in the way.
The difference is significant and trivial
The interval for the difference runs from about 0.21 to 1.59 marks out of 100.
A gap that small would not change anyone’s grade, and would not justify any policy. Yet it is “statistically significant”, because 680 students is a large enough sample to detect very small differences reliably.
Statistical significance answers “is this difference bigger than noise?”
It does not answer “is this difference big enough to matter?” That is a question about the subject, not about the statistics, and no \(p\)-value speaks to it.
Always read the interval, in the units of the problem, before deciding whether a result is important.
The groups were not assigned
The deeper problem is that nobody allocated students to high and low attendance. They chose, and the reasons they chose are not in the dataset.
A student who attends regularly may be more motivated, better prepared, less burdened by paid work or family obligations, or living closer to campus. Any of those could raise examination scores on its own. The two groups differ in attendance and in everything that attendance is correlated with, and the test cannot separate them.
The two-sample test compares two groups as they arrive. It has nothing to say about why they came to be different.
If the groups formed themselves, a difference in outcomes may reflect the treatment, or the reasons for selection, or both — and the arithmetic is identical either way. This is the problem of selection, and it is the central obstacle in all of empirical economics.
It is worth being precise about what has and has not been shown here.
We have shown that students who attended more scored slightly higher. That is a fact about the world and the test establishes it properly. We have not shown that attending more causes higher scores, and no amount of additional care with the \(t\) statistic would help.
What would help is a different design. If attendance had been assigned by something unrelated to the students themselves — a lottery, a timetable clash, a bus route — then the groups would be comparable in expectation and the difference could be read as an effect. Creating such comparisons, or finding them in the world, is the subject of the second part of this book.
The question “do these groups differ?” is a statistical one, and this unit answers it.
The question “does this treatment work?” is a question about how the groups were formed. It is answered by the design of the study, not by the choice of test.