4.3 What a p-value actually says
We already computed a probability by simulation. What exactly was it?
Back in Section 4.1 we simulated fifty thousand samples from a population whose mean really was 165 cm. Only about 2.4% of those samples produced a sample mean at least as far from 165 as the one we actually observed.
Without realising it, we had already computed a \(p\)-value.
A \(p\)-value is the probability of obtaining a sample at least as extreme as the one we observed, assuming the null hypothesis is true.
Notice the order carefully.
- We begin by assuming the null hypothesis is true.
- We ask what kinds of samples we would expect under that assumption.
- We compare our observed sample with those possibilities.
- The \(p\)-value is the proportion of those hypothetical samples at least as extreme as ours.
In the height example the \(p\)-value was about 0.024. If the true population mean really were 165 cm, a sample as unusual as ours would occur only about 2.4% of the time. That makes the observed sample unusual under the null hypothesis, which is why we would reject it at the 5% significance level.
The \(p\)-value does not tell us whether the null hypothesis is true. It tells us how surprising our data would be if the null hypothesis were true. This distinction is one of the most important ideas in statistics.
A small \(p\)-value means the observed data would be difficult to explain if the null hypothesis were correct. That is evidence against the null.
A large \(p\)-value means the observed data are quite ordinary under the null hypothesis, so we have no convincing reason to reject it. Notice what that does not mean: it does not prove the null hypothesis true. It means only that our data are compatible with it.
| \(p\)-value | Read it as | At \(\alpha = 0.05\) |
|---|---|---|
| 0.60 | A result like this is very common if the null hypothesis is true. | Fail to reject \(H_0\) |
| 0.18 | This would occur roughly once every six samples. Not unusual. | Fail to reject \(H_0\) |
| 0.049 | This would occur about once every twenty samples. Unusual enough to reject at the 5% level. | Reject \(H_0\) |
| 0.0001 | This would occur about once every ten thousand samples. Extremely unusual under the null. | Reject \(H_0\) |
Four ways to read it backwards
\(p\)-values are misquoted more often than any other number in empirical work, and the errors are predictable enough to be worth naming.
“The \(p\)-value is the probability that the null hypothesis is true.” False. The null hypothesis is assumed true in order to compute the \(p\)-value at all. A quantity computed under an assumption cannot also be the probability that the assumption holds.
“The \(p\)-value is the probability that we have made the wrong decision.” False. It measures how unusual the observed data are under the null, not the probability that our conclusion is incorrect.
“A large \(p\)-value proves the null hypothesis.” False. It means the data do not provide strong evidence against it, which is a much weaker statement.
“A very small \(p\)-value means the effect is important.” False. Statistical significance and practical importance are different ideas. A tiny effect produces a very small \(p\)-value if the sample is large enough.
A useful way to hold the idea:
A \(p\)-value measures how surprising the data are, not how believable the null hypothesis is.