4.4 How far is too far?

A gap between the sample and the claim is guaranteed. How large a gap is large enough to act on?

By now we know the basic logic. We begin by assuming the null hypothesis is true and ask: if that assumption were correct, how unusual would our sample be?

Section 4.1 answered that by simulation. Sampling repeatedly from a population whose mean really was 165 cm, a sample mean at least 3 cm away turned up about 2.4% of the time.

That raises a natural question. How unusual is unusual enough?

We need a rule telling us when the difference between the sample and the null is large enough to reject the claim.

Write \(\mu_0\) for the value the null claims, so that \(H_0: \mu = \mu_0\). If the sample mean lies close to \(\mu_0\), we treat the difference as ordinary sampling variation and keep the null. If it lies far enough away, we reject. That is, we reject whenever

\[|\bar{x} - \mu_0| \geq c\]

for some cutoff \(c\) we have yet to determine. The entire problem has reduced to a single question: how should we choose \(c\)?

Choosing the cutoff

We choose it before collecting any data, by deciding how much risk we are willing to run of rejecting a null hypothesis that is in fact true.

That risk is the significance level, written \(\alpha\). It is the probability of rejecting a true null hypothesis, and it is chosen in advance. Common choices are 0.05 and 0.01.

So \(c\) is chosen so that, if the null really is true, only a proportion \(\alpha\) of all possible samples would fall beyond it:

\[P\big[\,|\bar{X} - \mu_0| \geq c\,\big] = \alpha\]

Everything needed to solve that equation is already in hand from the previous units.

When the null is true, the Central Limit Theorem tells us the sampling distribution of the sample mean is approximately normal, centred on \(\mu_0\) with standard deviation \(\sigma/\sqrt{n}\):

\[\bar{X} \;\sim\; N\!\left(\mu_0,\; \frac{\sigma^2}{n}\right)\]

Following the usual convention, the second argument is the variance, so the standard deviation is its square root, \(\sigma/\sqrt{n}\) — the standard error.

Standardising, as in Section 3.4, removes the units and the sample size together:

\[Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}} \;\sim\; N(0, 1)\]

Dividing both sides of the probability statement by the standard error therefore gives

\[P\left[\,|Z| \geq \frac{c}{\sigma/\sqrt{n}}\,\right] = \alpha\]

The standard normal is symmetric, so that probability is split equally between the two tails:

\[P\left[\,Z \geq \frac{c}{\sigma/\sqrt{n}}\,\right] = \frac{\alpha}{2}\]

The point leaving an upper-tail probability of \(\alpha/2\) is the critical value, written \(Z_{\alpha/2}\). Since \(P[Z \geq Z_{\alpha/2}] = \alpha/2\) by definition, the bracketed quantity is \(Z_{\alpha/2}\):

\[\frac{c}{\sigma/\sqrt{n}} = Z_{\alpha/2} \qquad\Longrightarrow\qquad c = Z_{\alpha/2}\,\frac{\sigma}{\sqrt{n}}\]

Substituting that cutoff back into the rejection rule, and dividing both sides by the standard error, produces the form used in practice:

\[\boxed{\;\frac{|\bar{x} - \mu_0|}{\sigma/\sqrt{n}} \;\geq\; Z_{\alpha/2}\;}\]

The quantity on the left is the test statistic, usually written \(Z_{calc}\). The quantity on the right is the critical value.

The decision rule

  1. Compute the test statistic.
  2. Compare it with the critical value.
  3. If the statistic is larger, reject the null hypothesis.

Notice what has happened. The complicated question

Is my sample unusually far from the null hypothesis?

has become the much simpler question

Is my test statistic larger than the critical value?

That substitution is the whole point of standardising. Every one-sample problem about means — whatever the units, whatever the sample size — now arrives at the same comparison against the same handful of numbers.

Those familiar numbers — 1.645, 1.96 and 2.576 — are simply critical values of the standard normal distribution. There is no need to consult a printed table; qnorm() returns them to as many decimals as you like.

alpha <- c(0.10, 0.05, 0.01)

# Two-tailed: we want the value leaving alpha/2 in the upper tail
qnorm(1 - alpha / 2)
#> [1] 1.645 1.960 2.576

These are exactly the values defining the rejection regions for the most commonly used significance levels. They are not arbitrary quantities to memorise. They are the points leaving 10%, 5% and 1% of the probability split equally between the two tails of the standard normal distribution.