3.8 What a confidence interval does not mean

Four statements about the interval we just built. Which of them are correct?

By now you know how to construct a confidence interval and how to interpret it through repeated sampling. Yet confidence intervals are among the most misunderstood ideas in statistics, and experienced researchers interpret them incorrectly often enough that the errors have names.

Consider the following four statements. Decide what you think of each before reading on.

Statement 1: the probability the mean is inside

There is a 95% probability that the population mean lies between 24.4 and 27.8.

Incorrect. This is the most common misconception.

Once the sample has been collected, the confidence interval is fixed. The population mean is also fixed. Either the interval contains it or it does not — there is no longer any randomness to attach a probability to.

The 95% refers to the method, not to this particular interval. If we repeatedly collected samples and constructed intervals in exactly the same way, about 95% of them would contain the true population mean.

The uncertainty lies in the sampling process, not in the value of the population mean.

Statement 2: the spread of the population

Ninety-five percent of the population lies within this confidence interval.

Incorrect.

A confidence interval is about the location of the population mean, not about the spread of individual observations.

Our interval for average attendance runs from 24.4 to 27.8 classes — a few classes wide. Individual students in the same data attended anywhere from almost none of their classes to nearly all of them.

The clearest way to see that these are different quantities is to watch them respond to sample size. As \(n\) increases the interval narrows, because the mean is estimated more precisely. The variability among individual students does not change at all.

Statement 3: two intervals that overlap

The confidence intervals for two groups overlap, so the groups are not significantly different.

Not necessarily.

Two confidence intervals can overlap even when the difference between the two population means is statistically significant.

The reason is straightforward. Each interval describes uncertainty about a single mean. Whether two groups differ depends on the uncertainty surrounding their difference, which is a different quantity with a different standard error.

Later in the book we construct confidence intervals for differences between means. Those intervals, and not visual overlap, are the correct basis for comparison.

Statement 4: a wider interval

A wider confidence interval means the estimate is worse.

Not necessarily.

A wider interval means the estimate is less precise, and precision is not the same thing as quality.

A carefully designed survey with a small sample may produce a wide interval simply because the available information is limited. That interval is an honest report of genuine uncertainty.

A poorly designed survey may produce a very narrow interval by repeatedly sampling the wrong population. Section 2.2 showed exactly this: Survey B, which interviewed only urban households, produced estimates with a smaller spread than the correct design while centring on a figure ₹6,800 too high. The interval was precise, and precisely wrong.

Confidence intervals measure uncertainty due to sampling variation. They do not detect bias arising from poor sampling, non-response, or measurement error.

A confidence interval tells us what the data can reasonably support about an unknown population parameter.

It does not give the probability that a particular interval contains the truth, it does not describe the spread of individual observations, it does not settle whether two groups differ by whether they overlap, and it does not guarantee that the data were collected well.

Understanding these limitations is as much a part of using the interval as knowing how to calculate it.