This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Introduction and Overview

Lesson 1 closed with the counterfactual framework: causal inference requires us to compare groups under different exposure conditions. This lesson asks the immediate next question: which people should be in those groups? Sampling sits at the foundation of every study you'll design in HSCI 341 because the choice of who ends up in the study determines almost everything else — the validity of the results, the generalisability of conclusions, and the bias inventory you spent HSCI 230 learning to recognise. The four content sections move from broad principles to concrete formulas: Section 1 sets up the population hierarchy, sampling frame, and the probability theory that makes inference from a sample possible; Section 2 covers types of error and non-probability sampling; Section 3 details the major probability sampling designs; Section 4 closes with how to analyse data from complex surveys and how to determine sample size in advance.

Census vs. Sample

When we conduct research, we need data from either all individuals in a population or a subset of them. The process of obtaining this data is called measurement.

In a census, every individual in the population is evaluated. In a sample, data are collected from only a subset. Sampling is generally more convenient and less costly than conducting a full census. Interestingly, even a census can be viewed as a kind of sample — it captures the population at one point in time, making it a "sample" of the population over time.

Descriptive vs. Analytic Studies

Samples support two fundamental types of studies:

Choosing between a census and a sample is the first decision; choosing between a descriptive and an analytic purpose is the second. Both decisions sit on top of an even more fundamental concept — the relationship among the different populations a single study touches.

Hierarchy of Populations

Understanding the different populations involved in a study is essential for evaluating validity. There are three key populations to consider, each nested inside the next. The diagram and accordion below define them in turn; the labels that appear next to the arrows (external validity, sampling frame, internal validity) are the technical vocabulary you will use to talk about how a sample inherits or loses information from the broader population it was meant to represent.

Validity: Internal and External

Internal validity refers to whether the study results are valid for members of the source population. It indicates whether the study obtained the "correct" answer for that population. Much of epidemiology is dedicated to methods that ensure internal validity.

External validity involves a subjective assessment of whether results can be generalized to the broader target population. It is generally easier to generalize results from analytic studies (which evaluate associations) than from descriptive studies (which estimate prevalence).

The Sampling Frame

The sampling frame is the list of all sampling units in the source population. Sampling units are the basic elements that will be sampled (e.g., households, individuals). A complete list of all sampling units is required for drawing a simple random sample, though some other methods do not require such a complete listing.

The vocabulary of populations and sampling frames is necessary but not sufficient. The deeper question is why we can learn about a whole population from a small sample at all. The answer comes from probability theory.

Probability Theory: Why Sampling Works

Every quantity we estimate from a sample — a prevalence, a mean, a risk ratio — is the value of a random variable. If we drew a different sample tomorrow, the value would be slightly different. Probability theory is the formal language we use to describe how those values vary, and it is what lets us turn a single sample into a defensible statement about a population.

Three concepts do most of the work in introductory biostatistics:

A second idea worth naming is independence. Two observations are independent when knowing the value of one tells you nothing about the value of the other. Independence is the assumption that lets a small random sample stand in for a much larger population — and it is the assumption most often violated in real public-health data (clustered households, repeated measures on the same person, contagion in infectious disease).

Types of Probability Distributions

A probability distribution describes the values a random variable can take and how likely each value is. Distributions split first into discrete (countable outcomes — 0, 1, 2 cases) and continuous (any value on a range — height, blood pressure, time). The handful below are the ones you will encounter again and again in public health.

Discrete distributions

Continuous distributions

Each distribution above describes the underlying behaviour of a single random variable. But when we draw a sample from a population, the quantity we typically care about is a summary of that sample — a mean, proportion, or rate. To make inferences from such summaries, we need one more layer of theory.

Sampling Distributions and the Central Limit Theorem

So far we have talked about distributions of individuals in a population. Now consider the distribution of a statistic — for example, the mean of a random sample of n people. Because each sample of size n would give a slightly different mean, the mean itself has a distribution. We call it the sampling distribution of the mean.

Two facts about that sampling distribution drive almost all of frequentist inference:

The CLT is the hidden engine behind the bell curve that shows up everywhere in statistics. It is the reason a 95% confidence interval can be written as estimate ± 1.96 · SE: the 1.96 comes from the Normal distribution that the CLT promises us applies to the estimate, even when we have no idea what shape the underlying population has. The theorem traces from de Moivre (1733) through Laplace's 1812 binomial approximation to Lyapunov's 1901 general proof (Central limit theorem, Wikipedia).

Introduction and Overview

Section 1 set up the population hierarchy and the probability theory behind sampling. Section 2 turns to two practical consequences. First: when we draw conclusions from a sample, we make systematic types of mistake (Type I and Type II errors), and these mistakes are quantifiable. Second: there are sampling strategies that abandon probability theory altogether — convenience samples, judgement samples, snowball samples — with predictable consequences for inference. Both halves of this section give you the vocabulary to evaluate when those approaches are acceptable and when they are not.

Types of Error

In any study based on a sample, the variability of the outcome, measurement error, and sample-to-sample variability all affect results. When making inferences based on sample data, they are subject to error. Within hypothesis testing in analytical studies, there are two key types of error:

The error framework above assumes a probability sample. The next subsection covers what happens when investigators forgo formal random selection altogether — and what that costs them.

Non-Probability Sampling

Samples drawn without an explicit method for determining each individual's probability of selection are known as non-probability samples. Whenever there is no formal process for random selection, the sample should be considered non-probability. Sample selection that is unrelated to the outcome of interest leaves inference intact, but selection that depends on unmeasured determinants of the outcome produces selection bias — a form of specification error formalised by Heckman (1979) and reviewed for hidden populations by Sudman & Kalton (1986). There are three main types:

Click each card to learn more:

Introduction and Overview

Section 2 closed by walking through what non-probability sampling looks like and why it forfeits inferential validity. Section 3 returns to probability sampling and details the major variants you'll encounter in real surveys. The six tabs below are not interchangeable — each design buys a different combination of cost, complexity, and statistical efficiency. Read the comparison table at the end of the section as a decision aid you can return to whenever you're choosing a design.

What Is a Probability Sample?

A probability sample is one in which every element in the population has a known, non-zero probability of being included. This implies that a formal process of random selection has been applied to the sampling frame. The key advantage is that probability samples allow for valid statistical inferences about the source population.

Types of Probability Sampling

Comparison of Sampling Methods

Introduction and Overview

Section 3 walked through the major probability designs. Real surveys (the CCHS being the canonical Canadian example) typically combine several of these — stratification at the top level, clustering within strata, multistage selection within clusters. That combination is precisely what makes the analysis non-trivial: a complex sample design demands a complex analysis. The first half of this section covers how to do that analysis correctly. The second half closes the loop by walking through how to determine sample size before the data are collected.

Analysing Complex Survey Data

When data come from a complex sampling design (involving stratification, weighting, or clustering), the analysis must account for these features. Ignoring them can lead to incorrect point estimates and underestimated standard errors.

The Design Effect (deff)

The design effect (deff) summarizes the overall impact of the sampling plan on precision. It is the ratio of the variance from the complex sampling design to the variance that would have been obtained from a simple random sample of the same size. The concept and the term were coined by Leslie Kish in his classic textbook Survey Sampling (Kish, 1965) and remain the standard summary statistic for complex-design efficiency.

Finite Population Correction (FPC)

When the proportion of the population sampled is relatively large (>10%), precision improves beyond what would be expected from an "infinite" population. The finite population correction adjusts the estimated variance downward:

Analysing data correctly is necessary but not sufficient. Just as important is making sure you collected enough data to begin with — an underpowered study cannot be rescued by clever analysis. The remainder of this section covers sample-size calculation.

Sample-Size Determination

Choosing the right sample size involves both statistical and non-statistical considerations. Non-statistical factors include available resources (time, money, personnel) and the nature of the sampling frame. Statistical considerations include:

Key Sample-Size Formulae

# PART A -- three probability sampling designs from a 1,000-row frame
set.seed(341)
N <- 1000
frame <- data.frame(id = 1:N,
                    province  = sample(c("BC", "AB", "ON", "QC"), N, replace = TRUE),
                    household = sample(1:300, N, replace = TRUE))

srs   <- frame[sample(N, 100), ]                        # simple random
strat <- do.call(rbind, by(frame, frame$province,
                          function(d) d[sample(nrow(d), 25), ])) # stratified
sel_hh <- sample(unique(frame$household), 30)
clust  <- frame[frame$household %in% sel_hh, ]               # cluster

c(SRS = nrow(srs), Stratified = nrow(strat), Cluster = nrow(clust))

# PART B -- design-corrected prevalence with the survey package
library(survey)
dat <- data.frame(province = sample(c("BC","AB","ON","QC"), 2000, replace = TRUE),
                  smoker   = rbinom(2000, 1, 0.18),
                  weight   = runif(2000, 800, 2200))
des <- svydesign(ids = ~1, strata = ~province, weights = ~weight, data = dat)

mean(dat$smoker)                                # naive (unweighted)
svymean(~smoker, design = des)                  # design-corrected
confint(svymean(~smoker, design = des))         # 95% CI

# PART C -- sample-size calculations + design-effect adjustment
power.prop.test(p1 = 0.15, p2 = 0.10,
                power = 0.80, sig.level = 0.05)        # two proportions
power.t.test(delta = 5, sd = 14,
             power = 0.80, sig.level = 0.05)           # two means (SBP)

n_srs <- 685; rho <- 0.45; m <- 6
ceiling(n_srs * (1 + rho*(m - 1)))                # cluster-adjusted n

Bringing It All Together

This lesson moved sampling from a vague intuition to a structured set of decisions. You worked through the hierarchy of populations — target, source, study — and the way it maps onto internal and external validity. From there you built up the probability machinery (sampling distributions, the central limit theorem, Type I and Type II error, power) that lets a sample stand in for the population it was drawn from.

The second half of the lesson made those ideas operational: when to use simple, systematic, stratified, cluster, or multistage probability sampling; when non-probability designs are defensible; how complex survey data must be weighted and clustered in analysis; and how a sample-size calculation actually gets done. Lesson 4 will turn from who you measure to how — the design of the questionnaires those samples respond to.

Final Knowledge Assessment

Complete the following 15-question assessment. A score of 100% is required to complete the lesson. You may retake the assessment as many times as needed.