1.7 A language for randomness
Why do we need a mathematical language?
Throughout this chapter we have described repeated sampling almost entirely in plain English. We have spoken about rolling dice, drawing surveys, computing averages and repeating experiments thousands of times. That language has helped us build intuition, but it is no longer sufficient. To develop statistical inference, we need a concise and precise way of expressing these ideas.
The first concept we need is that of a random variable.
Random variables
Some quantities are known before we observe them. If you measure your height today, it is a fixed number. It may be unknown to you, but it is not changing from one repetition to the next.
Other quantities are different. Roll a die and you do not know what number will appear. Randomly select a worker from a population and you do not know that person’s earnings. Draw a sample of one hundred households and you do not know what the sample mean will be until the sample has actually been collected.
Statisticians describe such quantities as stochastic. A stochastic quantity is one whose value is determined by the outcome of a random process.
The mathematical object used to describe a stochastic quantity is called a random variable. We usually denote random variables by capital letters such as \(X\), \(Y\) and \(Z\). For example, \(X\) may denote the earnings of a randomly selected worker, \(Y\) the annual rainfall in a particular district, and \(D\) the outcome of rolling a fair die.
The sample mean deserves special attention. Every time we draw a different sample, we obtain a different average. The sample mean is therefore itself a random variable. Before observing the data, we denote it by \(\bar{X}\).
This observation is fundamental to inferential statistics. Since \(\bar{X}\) is a random variable, it has a probability distribution. That distribution tells us how the sample mean behaves across repeated samples. Once we understand its centre and spread, we can quantify the uncertainty associated with the single sample we actually observe.
Realisations
Eventually every random variable takes a particular observed value.
Suppose \(X\) represents the outcome of rolling a fair die. Before the die is rolled, \(X\) could take any value from one to six. After the die lands showing four, we have observed one realisation of the random variable, written as \(x = 4\).
The same distinction applies to the sample mean. Before collecting a sample, the sample mean is the random variable \(\bar{X}\). After collecting the data and computing the average, we observe one realisation of that random variable, written as \(\bar{x}\).
Throughout the remainder of this book, capital letters denote random variables, while lower-case letters denote their observed values.
The long-run average
Every random variable has a centre. The centre is called its expected value and is written as
\[E(X) = \sum_i x_i \, P(X = x_i).\]
The term expected value can be misleading. It is not the value we expect to observe on any particular repetition. Rather, it is the average value the random variable would take over a very large number of repetitions.
For a fair die,
\[E(X) = \frac{1+2+3+4+5+6}{6} = 3.5.\]
Of course, no single roll can ever produce 3.5. It is a theoretical average, not an individual outcome.
This distinction becomes clearer when we compare theory with data.
#> [1] 3.514#> [1] 3.5The close agreement between these two numbers is exactly what the Law of Large Numbers predicts. The expected value is calculated from the probability model, while the sample average is calculated from observed data. As the number of observations grows, the latter approaches the former.
When the random variable represents a randomly selected member of a population, its expected value is simply the population mean. For this reason we denote both by the same symbol,
\[E(X) = \mu.\]
Measuring variation
Knowing the centre of a distribution is not enough. Two random variables can have the same expected value but very different amounts of variation.
The spread of a random variable around its expected value is measured by its variance,
\[\operatorname{Var}(X) = E\big[(X - \mu)^2\big],\]
and the square root of the variance is the standard deviation,
\[\sigma = \sqrt{\operatorname{Var}(X)}.\]
The expected value tells us where a random variable is centred. The variance and standard deviation tell us how much it fluctuates around that centre.
Parameters and statistics
Statistics is concerned with learning about populations from samples. It is therefore essential to distinguish between quantities that describe a population and quantities computed from a sample.
A parameter is a numerical quantity describing the population. The population mean is written \(\mu\), the population variance \(\sigma^2\), and the population standard deviation \(\sigma\). These quantities are fixed — they do not change — but they are usually unknown, because we rarely observe the entire population.
A statistic, by contrast, is computed from a sample. The sample mean, sample variance and sample standard deviation are all statistics. Unlike parameters, statistics change from one sample to the next, because they depend on which observations happen to be selected.
The entire objective of inferential statistics can now be expressed in a single sentence:
Use statistics computed from a sample to learn about unknown parameters describing the population.
Everything that follows in this book — estimation, confidence intervals, hypothesis testing and regression analysis — is an extension of this single idea.
A summary of notation
The notation introduced in this section will be used throughout the book.
| Symbol | Meaning |
|---|---|
| \(X,\ Y,\ Z\) | random variables — quantities not yet observed |
| \(x,\ y,\ z\) | realisations — values actually observed |
| \(\bar{X}\) | the sample mean, before the sample is drawn |
| \(\bar{x}\) | the sample mean, once the sample is observed |
| \(n\) | the sample size |
| \(E(X)\) | expected value: the long-run average of \(X\) |
| \(\operatorname{Var}(X)\) | variance: the spread of \(X\) about its expected value |
And, separating what describes the population from what is computed from a sample:
| Quantity | Population (parameter) | Sample (statistic) |
|---|---|---|
| Mean | \(\mu\) | \(\bar{x}\) |
| Variance | \(\sigma^2\) | \(s^2\) |
| Standard deviation | \(\sigma\) | \(s\) |
The left-hand column is fixed and unknown. The right-hand column is observed and changes with every sample. Almost all of statistics is the effort to say something about the first using only the second.