HSCI 410 — Lesson 7

Modelling Count and Rate Data

Exploratory Data Analysis For Epidemiology

Kiffer G. Card, PhD, Faculty of Health Sciences, Simon Fraser University

Learning objectives for this lesson:

  • Distinguish among simple counts, rates with person-time denominators, population rates, and area-based counts
  • Describe the Poisson distribution and its mean=variance property
  • Specify and interpret a Poisson regression model including the offset term
  • Interpret incidence rate ratios (IR) from exponentiated Poisson coefficients
  • Evaluate Poisson models using Pearson, deviance, and Anscombe residuals
  • Distinguish apparent from real overdispersion and apply appropriate corrections
  • Compare negative binomial regression models (NB-1, NB-2) to Poisson regression
  • Apply zero-inflated, hurdle, and zero-truncated models to handle excess zeros

This course was developed by Kiffer G. Card, PhD, as a companion to Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.

Reference

Glossary — Key Terms, People & Concepts

📚 Reference page — available throughout the lesson

This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Key Concepts & Ideas
Count Data Non-negative integer outcomes (0, 1, 2, …) representing the number of events. Examples: cases of disease, hospital visits, deaths.
Rate Data Counts standardised by exposure or population time-at-risk (e.g., cases per 1000 person-years). Modeled as count + log(time) offset.
Exposure / Person-Time The denominator in a rate — how much “at-risk time” produced the counted events (e.g., person-years, animal-months). Enters the model as an offset.
Offset A predictor in a Poisson/NB model whose coefficient is fixed at 1 (typically log(person-time)). Converts a count regression into a rate regression.
Overdispersion Observed variance exceeds the mean, violating the Poisson mean = variance assumption. Inflates Type I error if ignored; addressed with negative binomial or quasi-Poisson.
Dispersion Parameter (φ) A parameter capturing the ratio of variance to mean in count models. φ = 1 indicates Poisson; φ > 1 indicates overdispersion.
Zero Inflation More zeros than the Poisson or NB distribution predicts. Suggests two underlying processes — one generating structural zeros and one generating counts.
Incidence Rate The number of new cases per unit person-time (e.g., per 1000 person-years). The natural quantity estimated by Poisson models with an offset.
Methods & Statistical Concepts
Poisson Distribution A discrete probability distribution for counts of rare independent events with a constant rate. Mean equals variance (λ).
Poisson Regression A GLM with Poisson family and log link: log(λ) = Xβ. Used for count or rate outcomes when the Poisson assumption holds.
Log Link Function The canonical link for count GLMs, log(λ) = Xβ. Coefficients are exponentiated to give rate ratios (multiplicative effects).
IRR (Incidence Rate Ratio) The exponentiated coefficient from a Poisson or NB rate model: IRR = exp(β). Interprets as the ratio of incidence rates per unit increase in the predictor.
Negative Binomial Regression A GLM that allows variance to exceed the mean by adding a dispersion parameter (k or θ). Standard remedy for overdispersed counts.
Quasi-Poisson A pragmatic fix for overdispersion that scales standard errors by √φ. Uses Poisson point estimates but inflates inference; doesn't yield a true likelihood.
Zero-Inflated Poisson (ZIP) A two-component model: a logistic regression for “structural zero” vs. “at-risk” status, and a Poisson for counts among the at-risk. Handles excess zeros.
Zero-Inflated Negative Binomial (ZINB) Like ZIP but with a negative binomial count component. Handles both excess zeros and overdispersion.
Hurdle Model A two-part model: a binary process for any-vs-no events, and a truncated count process for those with at least one event. Conceptually distinct from zero-inflation.
Vuong Test A test for comparing non-nested models (e.g., Poisson vs. zero-inflated Poisson). Evaluates which fits the observed data better.
Deviance / Pearson Residuals Residuals scaled appropriately for GLMs. Used to diagnose lack of fit, outliers, and overdispersion in Poisson and NB models.
Key People
Siméon-Denis Poisson (1781–1840) French mathematician and physicist who derived the Poisson distribution as a limit of the binomial. The distribution and the regression bear his name.
Norman Breslow (1941–2015) American biostatistician who developed Poisson-regression methods for cohort and rate data, and (with Day) authored the foundational two-volume Statistical Methods in Cancer Research.
Diane Lambert American statistician who introduced zero-inflated Poisson regression in a 1992 paper on manufacturing defects. The method is now standard for excess-zero counts.
No matching entries. Try a different search term.
Section 1

Introduction & The Poisson Distribution

⏱ Estimated time: 15 minutes

Introduction and Overview

Lessons 3–6 covered regression for continuous, binary, and ordered/multi-category outcomes. Lesson 7 takes on the next major outcome type: counts. Number of disease cases, hospital admissions, parasites, or events per unit time are all counts, and they need their own GLM family (Nelder & Wedderburn, 1972). The four content sections walk through this in order: the Poisson distribution and why it's the natural starting point (Section 1), Poisson regression with the offset term that lets us model rates as well as counts (Section 2), residuals and the most consequential complication — overdispersion (Section 3), and finally negative binomial and zero-adjusted models when standard Poisson breaks down (Section 4). The offset term is the key conceptual bridge to the rate-based designs you met in HSCI 230 Lesson 5 and HSCI 341 Lesson 5.

Learning Objectives

  • Distinguish simple counts, person-time rates, population rates, and area-based counts in epidemiological data.
  • State the assumptions of the Poisson distribution, including the variance-equals-mean property.
  • Compute Poisson probabilities and recognise the shape of the distribution at small versus large means.
  • Identify when count data are likely to violate Poisson assumptions and why it matters.

Why Model Count and Rate Data?

Many outcomes in epidemiology are measured as counts—the number of disease cases in a region, the number of doctor visits per year, or the number of parasites on a host animal. These outcomes differ fundamentally from continuous outcomes (modelled with linear regression) and binary outcomes (modelled with logistic regression). Count data require their own family of statistical models because they are discrete, non-negative, and often right-skewed (Coxe, West, & Aiken, 2009).

Chapter 18 introduces the statistical tools for modelling count and rate data, beginning with the Poisson distribution and Poisson regression, and extending to negative binomial and zero-adjusted models for situations where the basic Poisson assumptions are violated.

Types of Count and Rate Data

Before selecting an analytical approach, it is essential to understand which type of count or rate data you are working with. There are four main types encountered in epidemiological research:

🔢
Simple Counts
Click to learn more
Rates (Person-Time)
Click to learn more
🌎
Population Rates
Click to learn more
🗺
Area-Based Counts
Click to learn more

The Poisson Distribution

The Poisson distribution is the foundational probability distribution for modelling count data. It describes the probability of observing a given number of events in a fixed interval of time or space, assuming events occur independently at a constant average rate (Wikipedia, n.d.-a).

▸ INTERACTIVE STORY — POISSON RAIN Open full screen ↗

From raindrops to per-minute counts to the Poisson PMF. Next ▶ advances scenes.

A 6-scene visualization of count and rate data: random events at constant rate λ, per-minute counting, the histogram filling, the Poisson PMF emerging, and the bridge to rates and rate ratios.

Poisson Probability Function (Eq 18.1)
p(Y = y) = μy × e−μ / y!

In this formula, Y is the count of events, μ (mu) is both the mean and the expected number of events, and e is the base of the natural logarithm (≈ 2.718). The factorial in the denominator (y!) ensures the probabilities are correctly normalised. The Poisson distribution is defined for non-negative integers: y = 0, 1, 2, 3, …

Key Property: Mean = Variance

The most important property of the Poisson distribution is that the mean equals the variance: E(Y) = Var(Y) = μ (Coxe et al., 2009). This single-parameter property means that as the expected count increases, so does the variability. This assumption is central to Poisson regression—and when it is violated (variance > mean), we have overdispersion, which requires alternative approaches covered in later sections.

The Poisson distribution is particularly useful for modelling rare events in large populations. When the probability of an event is small and the number of trials (or opportunities) is large, the Poisson distribution provides an excellent approximation. Examples include the number of rare disease cases in a large population, or the number of equipment failures over an extended operating period.

When is the Poisson distribution appropriate?

The Poisson distribution is appropriate when: (1) events are independent of one another; (2) the rate at which events occur is constant over the observation period; (3) two events cannot occur at exactly the same instant; and (4) the probability of an event in a short interval is proportional to the length of the interval. In practice, these assumptions are often approximately met in epidemiological settings.

Shape of the Poisson distribution

When μ is small (e.g., μ < 3), the distribution is noticeably right-skewed—most observations cluster near zero with a long right tail. As μ increases, the distribution becomes more symmetric and begins to resemble a normal distribution. By the time μ ≥ 20, a normal approximation with mean μ and variance μ is often adequate.

Knowledge Check — Section 1

1. Which type of count data uses person-time in the denominator?

Rates use accumulated person-time at risk as the denominator. This is essential when subjects have varying amounts of follow-up time in a study.

2. What is the key property of the Poisson distribution?

The Poisson distribution has the property that E(Y) = Var(Y) = μ. This single-parameter property is central to Poisson-based models.

3. The Poisson distribution is most appropriate for modelling:

The Poisson distribution models counts, particularly when events are rare relative to the population at risk. It is defined for non-negative integers and is well-suited for rare event data.

Reflection

How might the type of count data (simple counts vs. rates) influence your choice of analytical approach in an epidemiological study you're familiar with?

Model answerSimple counts assume each unit of observation contributed the same exposure time — appropriate when follow-up is uniform (e.g., hospital admissions per week in a closed cohort). Rates explicitly divide counts by person-time, allowing comparisons across units with different exposure times — essential when follow-up is variable (cohort with censoring, repeat measurements, recurrent events). Example: simple Poisson works for daily emergency-department visit counts; rate-based Poisson (with offset(log(person_years))) is required for a chronic-disease registry where each patient contributes different person-time. Confusing the two leads to substantial bias: treating rate data as counts ignores the denominator, while treating count data as rates over-corrects for non-existent variation in exposure time.
Reflection saved!
* Complete the quiz and reflection to continue.
Section 2

Poisson Regression Model & Interpretation

⏱ Estimated time: 20 minutes

Introduction and Overview

Section 1 introduced the Poisson distribution as the probability model behind count data. Section 2 turns the distribution into a regression. The log-linear formulation (Nelder & Wedderburn, 1972) lets us link counts to predictors on a multiplicative scale, and the offset term is the trick that converts a count model into a rate model — precisely what you need to handle person-time denominators from the rate-based cohort designs you saw in HSCI 230 Lesson 5 and HSCI 341 Lesson 5.

Learning Objectives

  • Write down a Poisson regression model with a log link and identify its linear predictor.
  • Use an offset term to convert a count model into a rate model with person-time denominators.
  • Interpret exponentiated Poisson coefficients as multiplicative rate ratios.
  • Translate a fitted Poisson regression into incidence rates and predicted counts for new covariate values.

The Expected Count

The starting point for Poisson regression is the relationship between the expected number of events and the underlying rate. If an individual (or group) is observed for n units of person-time and the event rate is λ (lambda), the expected count is:

Expected Count (Eq 18.2)
E(Y) = nλ

Here, n represents the person-time at risk (e.g., person-years of follow-up) and λ is the incidence rate. The expected count is simply the product of the time at risk and the rate at which events occur. Different subjects may contribute different amounts of person-time, which must be accounted for in the model.

The Log-Linear Model

Poisson regression uses a log link function to relate the expected count to a linear combination of predictors (Wikipedia, n.d.-b). Taking the natural logarithm of both sides of the expected count equation and incorporating predictors gives us the Poisson regression model:

Poisson Regression Model (Eq 18.3)
ln(E(Y)) = ln(n) + β0 + β1X1 + β2X2 + …

The term ln(n) is the offset—a fixed term in the model that is not estimated but rather included to account for the fact that different observations may have different amounts of exposure (person-time). The β coefficients describe how the log of the expected count (or rate) changes with the predictors.

The Offset Term

The offset is one of the most important concepts in Poisson regression. It transforms the model from one that predicts counts to one that effectively predicts rates (Coxe et al., 2009).

Modelling Counts (Without Offset)

When no offset is included, the model predicts the expected count directly:

ln(E(Y)) = β0 + β1X1 + …

This is appropriate when all observations have the same amount of exposure or follow-up time. For instance, if all herds are observed for exactly one year, the count of disease cases directly reflects the rate. In practice, this situation is relatively uncommon—most epidemiological studies have subjects with varying follow-up times.

Modelling Rates (With Offset)

When the offset ln(n) is included, the model effectively predicts the rate rather than the raw count:

ln(E(Y)) = ln(n) + β0 + β1X1 + …

This is equivalent to modelling ln(E(Y)/n) = β0 + β1X1 + …, where E(Y)/n is the expected rate. The offset accounts for the fact that subjects with longer follow-up times are expected to accumulate more events simply by virtue of being observed longer. This is the standard approach when follow-up times vary across subjects.

Interpreting Poisson Regression Coefficients

In Poisson regression, the exponentiated coefficient eβ is interpreted as an incidence rate ratio (IRR) (Wikipedia, n.d.-d). This is analogous to the odds ratio in logistic regression but applies to rates rather than odds.

Incidence Rate Ratio (IRR)

For a one-unit increase in the predictor X1, the incidence rate is multiplied by eβ1. If β1 = 0.30, then IRR = e0.30 = 1.35, meaning the rate increases by 35% for each one-unit increase in X1. An IRR > 1 indicates an increased rate; an IRR < 1 indicates a decreased rate; and an IRR = 1 indicates no association.

Epidemiological Example: Mastitis in Dairy Herds

Suppose we model the number of mastitis cases per herd over one year, with herd size as a predictor and cow-years at risk as the offset. The Poisson regression yields βherd size = 0.012.

Interpretation: e0.012 = 1.012, so for each additional cow in the herd, the incidence rate of mastitis increases by 1.2%. A herd with 100 more cows would have an expected rate ratio of e0.012×100 = e1.2 = 3.32 compared to the baseline—a 3.32-fold higher mastitis rate.

R Activity — Poisson with an offset and a negative-binomial fall-back

The companion dataset phaa_followup.csv records how many GP visits each participant had during their follow-up. Because follow-up time varies, we need an offset of log(fu_years) to turn the count into a rate. The full annotated script is in r-activities/HSCI_410_Lesson_7_Count_and_Rate_Data.R.

library(MASS);  library(AER)
phaa <- read.csv("phaa_followup.csv", stringsAsFactors = FALSE)
phaa$smoker <- factor(phaa$smoker, levels = c("No","Yes"))

# 1. A peek at the count outcome
summary(phaa$gp_visits)
hist(phaa$gp_visits, breaks = 30,
     main = "GP visits during follow-up", xlab = "Visits")

# 2. Poisson regression with offset to model the RATE per person-year
fit_rate <- glm(gp_visits ~ age + smoker + hypertension
                            + offset(log(fu_years)),
                family = poisson, data = phaa)
summary(fit_rate)
exp(coef(fit_rate))                              # incidence-rate ratios
exp(confint(fit_rate))

# 3. Goodness of fit: Pearson chi^2 / df ~ 1 = good
sum(residuals(fit_rate, type = "pearson")^2) / fit_rate$df.residual

# 4. Formal overdispersion test
dispersiontest(fit_rate)

# 5. Negative binomial fall-back when Poisson is overdispersed
fit_nb <- glm.nb(gp_visits ~ age + smoker + hypertension
                              + offset(log(fu_years)), data = phaa)
AIC(fit_rate, fit_nb)
cbind(Poisson = exp(coef(fit_rate)),
      NegBin  = exp(coef(fit_nb)))

Why the offset isn't a predictor. Because its coefficient is fixed at 1, offset(log(fu_years)) shifts the intercept onto the rate scale without using a degree of freedom. The IRR for smokerYes tells you the multiplicative effect of smoking on the visit rate, holding age and hypertension constant. If dispersiontest() is significant, prefer glm.nb(); CIs widen but the point estimates are usually similar.

R Reflect on what you just ran

Use the questions below to interpret the output you produced. Look at your console / plot before answering.

1. From exp(coef(fit_rate)) and exp(confint(fit_rate)), report the IRR (and 95% CI) for smokerYes. Translate it into one sentence: how does the rate of GP visits per person-year differ between smokers and non-smokers?

Model answerexp(coef(fit_rate)) typically returns an IRR for smokerYes of about 1.40, 95% CI roughly (1.25, 1.55). Interpretation: smokers have an incidence rate of GP visits per person-year about 40% higher than non-smokers, holding other covariates constant. The CI excludes 1, so the effect is statistically significant.

2. Compute sum(residuals(fit_rate, type = "pearson")^2) / fit_rate$df.residual. Is it close to 1, or much larger? Also report the p-value from dispersiontest(fit_rate). What do those two pieces of evidence say about overdispersion?

Model answersum(residuals(fit_rate, type="pearson")^2) / fit_rate$df.residual typically returns a value around 1.4–2.0 — meaningfully larger than 1, suggesting overdispersion. dispersiontest(fit_rate) gives a small p-value (typically < 0.01), confirming statistically significant overdispersion. Both pieces of evidence point to the same conclusion: the Poisson assumption (variance = mean) is violated, and the standard errors from Poisson are likely too small (CIs too narrow, p-values inflated).

3. From AIC(fit_rate, fit_nb) and the side-by-side IRR comparison, which model do you prefer? Are the point estimates similar between Poisson and NegBin? What typically changes when you switch (point estimates, CIs, or both)?

Model answerAIC(fit_rate, fit_nb) typically shows AIC much lower for the negative binomial model (often hundreds of points lower), strongly favouring NegBin. The point estimates for IRRs are very similar between Poisson and NegBin — that's expected because both are estimating the conditional mean structure. What changes are the standard errors: NegBin SEs are larger (because they account for the extra variance from the over-dispersion parameter), so the CIs widen and p-values become less extreme. The lesson: overdispersion doesn't bias the point estimates much, but mis-specifying it gives over-confident inference.
Saved.

Poisson Regression for Relative Risk Estimation

An important application of Poisson regression is estimating relative risks (RR) directly from binary outcome data. When the outcome is rare, the Poisson model can provide estimates of the RR that are more interpretable than the odds ratios from logistic regression. This approach typically uses robust (sandwich) standard errors to account for the fact that binary data do not truly follow a Poisson distribution.

Knowledge Check — Section 2

1. What is the purpose of the offset term in Poisson regression?

The offset (log of person-time) accounts for the fact that subjects may have different amounts of follow-up or exposure time. It transforms the model from predicting counts to effectively predicting rates.

2. The exponentiated Poisson regression coefficient (eβ) is interpreted as:

eβ from Poisson regression represents the incidence rate ratio—the multiplicative change in the rate for a one-unit change in the predictor.

3. Using Poisson regression to estimate relative risks from binary data is appropriate when:

When the outcome is rare, the Poisson model can be used to estimate relative risks directly, which are more interpretable than odds ratios from logistic regression.

Reflection

Consider a study where participants have very different follow-up times. How would using an offset term change your interpretation compared to simply modelling raw counts?

Model answerThe offset term offset(log(person_years)) in a Poisson model fixes the coefficient on log(person-years) at exactly 1, effectively modelling the rate (events per person-year) rather than the raw count. Without an offset, the model would estimate a coefficient on log(person-years) that absorbs the relationship between exposure time and outcome — biased and not interpretable. With the offset, the IRRs read as multipliers on the rate (smokers have 1.4-fold higher rate of events per unit time), which is the unit of inference most epidemiologic questions want. Comparing to a count model without offset: a participant with 5 events in 1 year is the same as one with 5 events in 5 years, which is obviously wrong; the offset corrects this.
Reflection saved!
* Complete the quiz and reflection to continue.
Section 3

Evaluating Poisson Models & Overdispersion

⏱ Estimated time: 20 minutes

Introduction and Overview

Section 2 fit the Poisson model. Section 3 turns to evaluating it. Poisson regression makes a strong assumption — that the variance equals the mean — that real count data frequently violate (Ver Hoef & Boveng, 2007). Overdispersion is the most common diagnosis you'll make on a count model, and addressing it is what Section 4 will be about. First, though, you need the residuals and goodness-of-fit tools to detect it.

Learning Objectives

  • Compute and interpret Pearson, deviance, and Anscombe residuals from a Poisson model.
  • Apply the deviance and Pearson chi-squared statistics as overall goodness-of-fit tests.
  • Define overdispersion in terms of the variance-to-mean ratio and explain why it inflates Type I error.
  • Choose between quasi-Poisson, scale-corrected, and negative binomial responses to overdispersion.

Residuals for Poisson Models

Just as in linear regression, residuals are the primary tool for evaluating how well a Poisson model fits the observed data. However, because the variance of a Poisson variable depends on its mean, raw residuals (observed − expected) are not directly comparable across observations. Several types of standardised residuals have been developed:

Pearson Residuals

Pearson residuals standardise the raw residual by dividing by the square root of the expected value:

Pearson Residual (Eq 18.6)
rP = (y − μ̂) / √μ̂

This accounts for the Poisson assumption that Var(Y) = μ. If the model fits well, Pearson residuals should have approximately mean 0 and variance 1. The sum of squared Pearson residuals follows an approximate χ² distribution and can be used as an overall goodness-of-fit test.

Deviance Residuals

Deviance residuals are based on the contribution of each observation to the overall model deviance (the log-likelihood ratio comparing the fitted model to a saturated model). They are defined as:

di = sign(yi − μ̂i) × √[2(yi ln(yi/μ̂i) − (yi − μ̂i))]

Deviance residuals tend to be more normally distributed than Pearson residuals, especially when some expected counts are small. This makes them preferable for normal probability plots and other diagnostic displays.

Anscombe Residuals

Anscombe residuals use a transformation of the observed counts designed to make the residuals as close to normally distributed as possible. They apply a cube-root transformation to both the observed and expected values. Anscombe residuals are particularly useful when checking the normality assumption of residuals in Poisson models, and they complement Pearson and deviance residuals in a thorough model evaluation.

Goodness of Fit

The overall fit of a Poisson model can be assessed using the sum of squared Pearson residuals, which approximately follows a χ² distribution with (n − p) degrees of freedom, where n is the number of observations and p is the number of estimated parameters. A significant test statistic suggests the model does not fit the data adequately.

An important diagnostic is the dispersion parameter, estimated as the sum of squared Pearson residuals divided by the residual degrees of freedom:

Dispersion Parameter Estimate
φ̂ = ΣrP² / (n − p)

Under the Poisson assumption (mean = variance), φ should equal 1. Values substantially greater than 1 indicate overdispersion; values less than 1 indicate underdispersion.

Understanding Overdispersion

Overdispersion—when the observed variance exceeds the Poisson-assumed variance—is one of the most common problems in count data modelling (Ver Hoef & Boveng, 2007). It is critical to distinguish between two types:

Warning: Interpreting Overdispersion

Before concluding that overdispersion is “real,” always investigate whether the model is correctly specified. Adding missing predictors, removing outliers, or modelling non-linear effects may resolve apparent overdispersion without needing to change the distributional assumptions. Applying overdispersion corrections to a misspecified model can mask important features of the data.

Apparent Overdispersion

Apparent overdispersion arises from problems with the model rather than the data-generating process itself. Common causes include:

  • Outliers: A few extreme observations can inflate the dispersion statistic dramatically.
  • Missing important predictors: If key covariates are omitted from the model, the unexplained variation appears as overdispersion.
  • Wrong model form: Using a linear predictor when the true relationship is non-linear.
  • Non-linear effects: Failing to include quadratic or other polynomial terms for predictors with curvilinear relationships.

Apparent overdispersion can be resolved by correcting the model specification—removing outliers, adding missing predictors, or using the correct functional form.

Real Overdispersion

Real overdispersion reflects genuine extra-Poisson variation in the data that cannot be explained by observable covariates. This often arises from:

  • Unobserved heterogeneity: Subject-level variation in the underlying rate that is not captured by measured predictors.
  • Clustering: Events within groups (e.g., animals within herds) are correlated, violating the independence assumption.
  • Biological variability: Inherent variation in susceptibility or exposure that exceeds what the Poisson model allows.

Real overdispersion requires statistical corrections such as scaling standard errors, using negative binomial regression, or employing random effects models.

Approaches to Handling Overdispersion

ApproachHow It WorksWhen to Use
Scale SEs by √φMultiplies standard errors by the square root of the estimated dispersion parameter; coefficients unchangedMild to moderate overdispersion; quick fix when coefficient estimates are trusted
Negative binomial regressionAdds an extra parameter (α) to model the excess variance explicitlyModerate to severe overdispersion; when a more principled model is desired
Random effects / GLMMIncludes subject- or group-level random intercepts to capture unobserved heterogeneityClustered data (e.g., animals within herds); hierarchical study designs
GEE (robust SEs)Uses generalised estimating equations with an empirical (sandwich) variance estimatorClustered data when marginal (population-averaged) estimates are of primary interest
Knowledge Check — Section 3

1. In a Poisson model, overdispersion is indicated when:

Overdispersion occurs when the observed variance exceeds the Poisson-assumed variance (mean), giving a dispersion parameter φ > 1.

2. Which of the following is NOT a cause of apparent overdispersion?

Inherent biological variation causes real (not apparent) overdispersion. Apparent overdispersion is caused by model misspecification issues such as outliers, missing predictors, or incorrect functional form.

3. One approach to handling real overdispersion is:

Negative binomial regression explicitly models the extra-Poisson variation by adding a parameter that allows the variance to exceed the mean. Removing outliers and adding predictors address apparent overdispersion instead.

Reflection

Why is it important to distinguish between apparent and real overdispersion before choosing a correction strategy? What could go wrong if you apply the wrong fix?

Model answerApparent overdispersion can arise from model misspecification — missing important covariates, missing interactions, wrong functional form for a continuous predictor — rather than from true heterogeneity in the rate. The fix depends on the cause: (a) if misspecification, add the missing covariates and re-check dispersion; the ‘overdispersion’ often resolves. (b) If real (true heterogeneity beyond what the linear predictor captures), switch to negative binomial or quasi-Poisson. Wrong fix consequences: applying NegBin to a misspecified Poisson model patches the SE inflation but leaves the biased point estimates from missing covariates; applying robust SEs instead may also misrepresent the true variability if the underlying structure has zero-inflation. Always investigate dispersion's source through residual diagnostics, plots of variance vs. mean, and inclusion of plausible missing covariates.
Reflection saved!
* Complete the quiz and reflection to continue.
Section 4

Negative Binomial & Zero-Adjusted Models

⏱ Estimated time: 20 minutes

Introduction and Overview

Section 3 named overdispersion as the most common problem with Poisson regression. Section 4 closes the lesson with the standard fixes: the negative binomial distribution (which adds a free dispersion parameter; Ver Hoef & Boveng, 2007), and zero-adjusted models for the special case where there are far more zeros than Poisson or negative binomial alone can accommodate (Lambert, 1992).

Learning Objectives

  • Derive the negative binomial as a Gamma-mixed Poisson and contrast NB-1 and NB-2 parameterisations.
  • Fit a negative binomial regression and compare it to a Poisson fit using a likelihood-ratio test on the dispersion parameter.
  • Identify zero-inflation versus zero-truncation and choose the appropriate zero-adjusted model.
  • Interpret a hurdle or zero-inflated model in terms of separate processes for the zero and the count components.

The Negative Binomial Distribution

The negative binomial (NB) distribution extends the Poisson by adding an extra parameter α that captures the additional variation not accounted for by the Poisson assumption. Conceptually, the NB distribution arises when the Poisson rate itself varies randomly across individuals—each subject has their own λ, drawn from a Gamma distribution (Wikipedia, n.d.-c).

The NB distribution allows the variance to exceed the mean, making it the natural first choice when overdispersion is present (Ver Hoef & Boveng, 2007). Two common parameterisations define how the variance relates to the mean:

NB-1: Linear Variance

NB-1 Variance Function (Eq 18.8)
Var(Y) = μ + αμ = μ(1 + α)

In the NB-1 parameterisation, the variance increases linearly with the mean. The overdispersion is proportional to the mean: doubling the expected count doubles the excess variance. The ratio Var(Y)/μ = (1 + α) is constant across all observations, making NB-1 similar to a quasi-Poisson model with a fixed dispersion parameter.

NB-1 is sometimes preferred when overdispersion is relatively constant across the range of predicted values. However, it is less commonly used in practice than NB-2.

NB-2: Quadratic Variance

NB-2 Variance Function (Eq 18.9)
Var(Y) = μ + αμ²

In the NB-2 parameterisation (the most commonly used form), the variance increases quadratically with the mean. Observations with higher expected counts have proportionally more overdispersion. This is often more realistic in biological settings where variability tends to grow faster than the average.

The NB-2 model is the default in most statistical software (e.g., Stata’s nbreg, R’s glm.nb()). When α = 0, the NB-2 model reduces to the Poisson model, making the Poisson a special (nested) case of NB-2.

Negative Binomial Regression

The NB regression model uses the same log-linear form as Poisson regression—the only difference is in the assumed distribution of the outcome:

NB Regression Model
ln(E(Y)) = ln(n) + β0 + β1X1 + β2X2 + …

Coefficients are interpreted identically to Poisson regression: eβ gives the incidence rate ratio. The key advantage is that the NB model produces correct standard errors even when overdispersion is present, because the extra variation is explicitly modelled through α.

Testing Poisson vs. Negative Binomial

Since the Poisson model is nested within the NB model (when α = 0), a likelihood ratio test (LRT) can be used to determine whether the NB model provides a significantly better fit. A significant LRT indicates that overdispersion is present and the NB model is preferred. Note that this is a boundary test (testing α = 0 vs. α > 0), so the p-value from the standard χ² reference distribution is conservative.

Zero-Adjusted Models

Standard count models (Poisson and NB) may not adequately handle datasets with an unusual number of zeros (Lambert, 1992). Three families of models have been developed to address different zero-related problems:

🔠
Zero-Inflated Models
Click to learn more
🏃
Hurdle Models
Click to learn more
Zero-Truncated Models
Click to learn more

Choosing Among Zero-Adjusted Models

The choice depends on the data-generating process:

  • If some zeros are “structural” (from a fundamentally different process) and others arise from the count process, use a zero-inflated model.
  • If the zero/non-zero distinction is a separate decision from the magnitude of the count, use a hurdle model.
  • If zeros are impossible by design, use a zero-truncated model.
ModelSource of ZerosKey FeatureTest / Comparison
Zero-InflatedBoth components (structural + count)Mixture of logistic + count modelVuong test vs. standard model
HurdleBinary component onlyTwo-part: binary then truncated countLRT or AIC/BIC comparison
Zero-TruncatedZeros cannot occurConditional on Y > 0Applied when sampling excludes zeros
Knowledge Check — Section 4

1. The NB-2 model differs from the Poisson model by:

The NB-2 model adds an αμ² term to the variance, allowing the variance to increase quadratically with the mean—this is the most commonly used negative binomial model.

2. Zero-inflated models are appropriate when:

Zero-inflated models handle situations where some zeros come from a separate process (structural zeros) in addition to zeros from the count process, resulting in more zeros than the standard count model predicts.

3. The key difference between a hurdle model and a zero-inflated model is:

In a hurdle model, ALL zeros come from the binary (logistic) part; once the hurdle is crossed, only positive counts are modelled. In zero-inflated models, zeros can come from both components.

Reflection

When might you choose a hurdle model over a zero-inflated model in practice? Think of an epidemiological example where the distinction matters.

Model answerHurdle and zero-inflated models both handle excess zeros but with different mechanisms. Hurdle: the zero process is independent of the count process — you first decide whether any event occurs (a binary process), and only if “yes” do you count events (typically a zero-truncated Poisson or NegBin). Zero-inflated: zeros come from two sources — a structural-zero process (always zero, e.g., never susceptible) plus the count process (could be zero or non-zero among the susceptible). Choose hurdle when the binary “any event” decision is conceptually separable from the count “how many” (e.g., GP visits: did you go at all this year? if yes, how many?). Choose zero-inflated when some subjects are structurally immune (e.g., contraception use among postmenopausal women — structural zeros — vs. premenopausal users counting cycles). Distinction matters because the two models give different interpretations of the ‘extra-zero’ subjects.
Reflection saved!
* Complete the quiz and reflection to continue.
Final Assessment

Lesson 7 — Comprehensive Assessment

⏱ Estimated time: 25 minutes

Bringing It All Together

This lesson assembled the count-and-rate toolkit. We started with the Poisson distribution and used its log-link form to build a regression, then introduced the offset term — the small piece of arithmetic that converts a count model into a rate model and unifies the cohort and incidence-density designs you have already met. From there we moved to evaluation: Pearson, deviance, and Anscombe residuals; goodness-of-fit tests; and the variance-to-mean comparison that makes overdispersion visible.

Section 4 introduced the standard responses to overdispersion. Negative binomial regression adds a dispersion parameter and is the everyday workhorse when Poisson assumptions break. Zero-adjusted models — hurdle and zero-inflated — handle outcomes where the data-generating process has separate components for whether an event occurs at all and how often it occurs given that it can. Together these models cover the bulk of count-data analyses you are likely to read or run.

The final assessment asks you to recognise count and rate situations on sight, reach for the right model, and report rate ratios with appropriate uncertainty. Use the diagnostic flowchart from Section 3 as your default answer to “which model?”.

Key Takeaways from Lesson 7

  • Counts and rates are a distinct outcome family with their own GLM — the Poisson, with a log link.
  • The offset term turns a count model into a rate model by absorbing person-time or population denominators.
  • Exponentiated coefficients in Poisson and negative binomial regression are rate ratios, interpreted multiplicatively.
  • Overdispersion — variance exceeding mean — is the most common Poisson failure mode and inflates Type I error if ignored.
  • Negative binomial regression is the default fix; use it whenever the dispersion test rejects Poisson.
  • Zero-inflated and hurdle models handle outcomes with a separate “at-risk” process generating excess zeros.

Final Reflection

Reflecting on the full range of count and rate models covered in this lesson, how would you decide which model to use for a new dataset? What diagnostic steps would you follow?

Model answerDecision flowchart: (1) Is the outcome a count? Yes → continue; no → use a different family. (2) Is exposure time variable? Yes → use offset(log(time)); no → simple Poisson. (3) Is the Poisson assumption (variance = mean) met? Check Pearson dispersion ratio and formal tests. If close to 1, Poisson is appropriate. If > 1.2, suspect overdispersion. (4) Investigate dispersion source: misspecification (missing covariates/interactions) vs. true heterogeneity. (5) If true heterogeneity, fit negative binomial; compare AIC to Poisson. (6) Are there excess zeros beyond what NegBin predicts? Check zero-inflation diagnostic plots. If yes, fit hurdle or zero-inflated based on substantive mechanism. (7) Verify final model with residual plots, predicted vs. observed marginal distributions, and out-of-sample prediction if possible. Document the decision path in the methods section.
Reflection saved!
Final Assessment — Lesson 7 (15 Questions)

1. Which type of count data involves dividing event counts by accumulated person-time?

Incidence rates use person-time at risk as the denominator, dividing the number of new events by the accumulated person-time of follow-up.

2. The Poisson distribution assumes:

The fundamental property of the Poisson distribution is that E(Y) = Var(Y) = μ. This single-parameter property is what distinguishes it from the negative binomial.

3. In Poisson regression, the offset term represents:

The offset is ln(person-time) which accounts for different exposure times across observations, allowing the model to estimate rates rather than raw counts.

4. The exponentiated coefficient from a Poisson regression (eβ) is interpreted as:

eβ gives the incidence rate ratio—the multiplicative change in the event rate per unit change in the predictor.

5. Pearson residuals for Poisson regression are calculated as:

Pearson residuals standardize the difference between observed and expected by dividing by √μ̂, which accounts for the Poisson variance assumption.

6. A dispersion parameter of 3.5 in a Poisson model suggests:

A dispersion parameter substantially greater than 1 indicates overdispersion—the data have more variability than the Poisson model assumes. A value of 3.5 is well above the expected value of 1.

7. Which is a cause of APPARENT (not real) overdispersion?

Missing important predictors is a model specification issue that creates apparent overdispersion, which can be resolved by correcting the model rather than changing distributional assumptions.

8. Scaling standard errors by √φ addresses overdispersion by:

Scaling SEs by √φ makes confidence intervals wider and P-values more conservative, but does not change point estimates of the coefficients.

9. The NB-2 model specifies the variance function as:

The NB-2 model has variance μ + αμ², which is quadratic in the mean and is the most commonly used form of negative binomial regression.

10. To test whether negative binomial regression provides a better fit than Poisson, you use:

Since Poisson is nested within NB (when α = 0), a likelihood ratio test comparing the two models tests whether the extra parameter is needed.

11. A zero-inflated Poisson model combines:

ZIP models have two components: a logistic model determining whether an observation is from the “always zero” group, and a Poisson model for the count process.

12. The Vuong test is used to:

The Vuong test is specifically designed to compare non-nested models, particularly zero-inflated models versus their standard counterparts.

13. In a hurdle model, zero counts are generated by:

In a hurdle model, ALL zeros come from the binary component. Once the “hurdle” is crossed (event > 0), only positive counts are modelled by the truncated count component.

14. Zero-truncated models are appropriate when:

Zero-truncated models are used when the sampling design excludes zeros—for example, hospital length of stay (minimum 1 day) or number of items purchased by customers.

15. Poisson regression can be used to estimate relative risks directly from binary outcome data because:

When outcomes are rare, Poisson regression with robust standard errors provides valid estimates of the relative risk, which is often more interpretable than odds ratios from logistic regression.

Lesson 7 Complete!

Lesson 8 — Survival Data — takes on the next major outcome type: time-to-event. Survival models extend the regression toolkit to handle censoring (people who haven't yet had the event) and the rich information in the timing of events, not just whether they occurred.

Congratulations! You have successfully completed the Modelling Count and Rate Data module. Your responses have been downloaded automatically.