Systematic Reviews and Meta-Analysis
Fundamental Epidemiological Concepts and Approaches
Kiffer G. Card, PhD, Faculty of Health Sciences, Simon Fraser University
Learning objectives for this lesson:
- Place systematic reviews and meta-analyses within the hierarchy of evidence, and explain why the pyramid is a heuristic, not a verdict
- Use the DIKW hierarchy (data → information → knowledge → wisdom) to frame the role of evidence synthesis in public-health decision-making
- Carry out the steps of a systematic review, from specifying the question to synthesising results
- Complete the data-extraction process to provide data suitable for meta-analysis
- Calculate summary estimates of effect and evaluate heterogeneity among study results
- Choose between fixed- and random-effects models and explain when each is appropriate
- Present and interpret forest plots and other graphical displays of meta-analysis results
- Evaluate potential causes of heterogeneity using subgroup analysis, stratification, and meta-regression
- Evaluate the potential impact of publication bias using funnel plots and related methods
- Determine if results have been influenced by an individual study (sensitivity analysis)
This course was developed by Kiffer G. Card, PhD, as a companion to Dohoo, I. R., Martin, S. W., & Stryhn, H. (2012). Methods in Epidemiologic Research. VER Inc.
Glossary — Key Terms, People & Concepts
📚 Reference page — available throughout the lesson
This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.
Hierarchy of Knowledge & Systematic Reviews
Introduction and Overview
Lesson 1 set up the foundations of epidemiology — its history, its ways of knowing, and the trust scaffolding (research integrity, reproducibility) that lets a body of evidence accumulate. Section 4 of that lesson made the case that empiricism is one way of knowing among several, and Section 11 left you with a personal stance on which of those ways you would lean on when the stakes are public-health decisions. Lesson 2 picks up at exactly that seam: once you commit to evidence-based reasoning, how do you tell stronger evidence from weaker? Before we open up the machinery of systematic reviews and meta-analyses, we need a shared way of organising the answer. That organising idea — the hierarchy of knowledge — is where this lesson starts, and it is the lens through which every later lesson in the series will be read.
The Hierarchy of Knowledge
Before reading further, sit with the prompt below for a moment. The rest of the section is easier to follow if you have already put your own intuitions on paper.
💭 Discussion Prompt — Rank the Evidence
Suppose four sources tell you the same intervention works:
- A trusted senior clinician's experience over a 30-year career.
- A single, well-written case report.
- A well-designed cohort study of 4,000 people followed for five years.
- A meta-analysis of twelve randomised trials covering 18,000 participants.
Rank them from weakest to strongest evidence for a public-health recommendation. Then ask: what is doing the ranking work — sample size? design? the number of independent studies? Could any single feature of one of the weaker sources overturn your ranking?
Epidemiologists answer this question with a layered model usually called the hierarchy of evidence, or simply the evidence pyramid. The pyramid is not a strict ranking of every study against every other — it is a heuristic for thinking about how much variation a particular design has ruled out. At the base sit forms of evidence that are easy to produce but hard to defend on their own: expert opinion, individual experience, single case reports. Moving up, we add structured comparison (case series → cross-sectional → case-control → cohort). Higher still are interventional designs that randomise the exposure (randomised controlled trials), which under ideal conditions rule out confounding entirely. At the apex sit the designs that synthesise across all of the layers below: systematic reviews and meta-analyses.
A Complementary Hierarchy: DIKW
The evidence pyramid tells us where to look. A second, complementary hierarchy tells us what the evidence is for. The DIKW pyramid — Data → Information → Knowledge → Wisdom — is widely used in informatics, decision science, and clinical practice:
- Data are raw observations: counts, measurements, dates, lab values.
- Information is data given structure: organised, contextualised, compared.
- Knowledge is information appraised: filtered for quality, integrated across sources, ready to defend in print.
- Wisdom is knowledge exercised in a specific context, under uncertainty, in service of a particular decision.
A meta-analysis is not the same thing as wisdom. It is the highest-order knowledge layer in the evidence pyramid — the most thoroughly appraised, most heavily aggregated form of knowledge the field knows how to produce. Wisdom is what a clinician, policy-maker, or community partner does with that knowledge in their particular case — and that step is irreducibly human, contextual, and value-laden. Lesson 1's call for a critical epidemiology that asks who benefits, who is harmed, whose priorities lives at the wisdom layer, not the knowledge layer. Keeping the two hierarchies side-by-side is a useful corrective against treating a forest plot as if it answered every question on its own.
Why Start at the Top of the Pyramid?
HSCI 230 spends Lessons 3–12 working through the rungs below the apex: case-control, cohort, ecological, and cross-sectional designs, plus the threats (selection, information, confounding) that compromise each. So why does Lesson 2 begin at the top? Three reasons, each of which shapes the rest of the series:
- You need the appraisal lens before you read any single study. When you encounter a primary observational study in Lesson 3 onward, the first question is not "is this study good?" but "what does the synthesised literature on this question already say, and how does this study fit in?" The synthesis layer is the lens through which every later study should be read.
- The hierarchy is a heuristic, not a verdict. A poorly conducted systematic review is weaker evidence than a well-designed cohort study. A well-designed case series of an unprecedented exposure may be the only evidence in existence. The hierarchy tells you where to look for the strongest available evidence — it does not exempt you from appraising what you find. Section 4 of this lesson will make that point concrete with publication bias, influential studies, and outcome-scale issues that can compromise even a textbook meta-analysis.
- Every tool in this lesson exists because the lower tiers can mislead. PROSPERO registration, PRISMA reporting, GRADE certainty ratings, and the Cochrane risk-of-bias appraisals (so named for Archie Cochrane, whose 1972 monograph Effectiveness and Efficiency framed the case for systematically appraised RCT evidence) — each is a field-wide response to a specific failure of unstructured evidence accumulation. Lesson 1's trust ledger and the open-science reforms in its later sections are the wider movement these tools belong to. The hierarchy of evidence is, in a real sense, a history of the lessons epidemiologists learned the hard way.
With the hierarchy in mind, the four content sections of this lesson move from broad to specific. Section 1 sets up systematic reviews as a structured way to identify and appraise all relevant studies; Section 2 turns to meta-analysis as the quantitative pooling of effect estimates, including the central choice between fixed- and random-effects models; Section 3 covers the forest plot and heterogeneity analysis; Section 4 closes with the threats to a meta-analysis that survive correct technique. You will revisit these tools throughout the remainder of HSCI 230 as you appraise individual observational studies — every appraisal sits inside the question, "and what does the synthesised literature say?"
Learning Objectives
- Place systematic reviews and meta-analyses within the hierarchy of evidence, and explain why the pyramid is a heuristic rather than a verdict on any single study.
- Articulate how the DIKW hierarchy (data → information → knowledge → wisdom) frames evidence synthesis in public-health decision-making.
- Distinguish a narrative review from a systematic review and explain why narrative reviews are unsuitable for guiding policy.
- Walk through the seven steps of a systematic review (Sargeant et al., 2006), from question specification through synthesis.
- Describe the role of the PRISMA reporting checklist and the Cochrane Risk of Bias tools in producing a trustworthy review.
- Identify the inclusion/exclusion criteria, search strategy, and quality-appraisal steps that make a systematic review reproducible.
16.1 Why Systematic Reviews?
When making decisions about health interventions, we want to use all available information. Unfortunately, the literature is often inconclusive and conflicting—individual studies may produce results ranging from statistically significant to inconsequential, and the variation among results may be greater than expected from chance alone. A classic demonstration of the cost of relying on narrative summaries is Antman et al. (1992), who showed that expert reviews of myocardial-infarction treatments lagged years behind what a cumulative meta-analysis of randomised trials had already established.
There are two fundamental approaches to formally reviewing available data: a narrative review and a systematic review (which may include a meta-analysis).
Narrative Reviews
In a narrative review, each study is considered individually, and the reviewer subjectively assesses the evidence. Narrative reviews have several limitations:
- They tend to be carried out by subject experts who may bring preconceived opinions, resulting in biased review
- They often lack a structured methodology for identifying and assessing relevant studies
- Small but well-designed studies may be omitted if they lack statistical power
- Inclusion criteria are often not described in adequate detail
- There is a tendency to weight all studies equally, when they should not all receive equal weight
Narrative reviews should only be used to provide an overview of literature, not to guide treatment or policy decisions.
Systematic Reviews
A systematic review uses a structured, transparent methodology to identify, evaluate, and synthesise all relevant studies on a specific question. It minimises bias and provides reproducible results. A systematic review may or may not include a quantitative meta-analysis, depending on the nature and quality of available data.
16.2 Steps of a Systematic Review
A systematic review follows seven key steps (Sargeant et al., 2006):
The question should be driven by a clinical or health-policy objective, not by data availability. It is often more desirable to address a broad question (e.g., the ability of β-blockers as a class to reduce myocardial infarction risk) rather than a narrow one (e.g., one specific drug), to enhance generalisability. The question should specify the intervention(s), outcomes, comparisons, and eligible study designs.
The review protocol should be objective and transparent—a reader should be able to duplicate it. This corresponds to the “Materials and Methods” of a primary study and covers all subsequent steps. A clear protocol minimises subjective decisions during the review process.
The literature search must be complete and well-documented. This involves searching major electronic databases (e.g., PubMed/Medline), reviewing reference lists of identified papers, and searching for grey literature (conference proceedings, theses, unpublished studies). The search strategy, databases, date ranges, and keywords must all be documented.
Inclusion criteria specify the intervention(s), population(s), outcome(s), and study types eligible for the review. Exclusion criteria may include language restrictions, publication date cutoffs, or accessibility. Relevance should be assessed independently by two or more reviewers using the title and abstract, followed by full-text review.
Each study’s internal and external validity must be evaluated. The Cochrane Collaboration’s risk-of-bias tool (now RoB 2 for randomised trials; Sterne et al., 2019) assesses domains including sequence generation, allocation concealment, blinding, incomplete data, and selective reporting; ROBINS-I (Sterne et al., 2016) provides the parallel framework for non-randomised studies. Quality assessment results can be used to exclude studies, weight them differentially, or evaluate quality as a source of heterogeneity.
From each study, you need the point estimate of the outcome and a measure of its precision (SE or CI). Data extraction should be carried out independently by two investigators using a standardised template, with any differences resolved by discussion. Watch for duplicate reporting of the same data in multiple publications.
Results can be summarised qualitatively (narrative description with tabular/graphical display) or quantitatively (meta-analysis). A quantitative meta-analysis computes a pooled summary estimate of the effect, weighted by the precision of each study, and investigates reasons for variation across studies.
16.2.1 Two Tools That Underpin a Trustworthy Review
Modern systematic reviews are expected to do more than simply follow the seven steps—they must do so transparently and they must communicate how confident readers should be in the conclusions. Two widely adopted tools have become the de facto standards for these expectations: PROSPERO for protocol registration and GRADE for rating the certainty of the evidence.
📝 Call-Out: PROSPERO — Prospective Protocol Registration
What it is. PROSPERO is the international prospective register of systematic reviews, hosted by the Centre for Reviews and Dissemination at the University of York. Reviewers submit their protocol—research question, eligibility criteria, search strategy, planned analyses, and outcomes—before beginning study selection or data extraction.
Why it matters. Prospective registration mirrors the role of ClinicalTrials.gov for primary trials. It creates a public, time-stamped record of the review’s plan, which:
- Reduces the risk of outcome-reporting bias and post hoc changes to inclusion criteria
- Helps avoid duplication of effort across research teams working on the same question
- Allows readers, peer reviewers, and editors to compare the final review against the planned protocol
PROSPERO registration is now required or strongly encouraged by Cochrane, by many journals (e.g., BMJ, JAMA, Annals of Internal Medicine), and by the PRISMA 2020 reporting guideline (Page et al., 2021; original statement: Moher et al., 2009). Pair the registration with PRISMA at submission and you have a fully transparent audit trail from question to conclusion.
🎯 Call-Out: GRADE — Rating the Certainty of the Evidence
What it is. The Grading of Recommendations, Assessment, Development and Evaluations (GRADE) framework (Guyatt et al., 2008) provides a structured method for rating how much confidence we should have in the body of evidence for each outcome of a systematic review—separately from rating the strength of any clinical recommendation that follows.
How it works. Each outcome starts at a baseline level of certainty based on study design (RCTs start at high; observational studies start at low), and is then downgraded or upgraded based on eight domains:
| Domain | Direction | What it captures |
|---|---|---|
| Risk of bias | Downgrade | Methodological limitations of the included studies |
| Inconsistency | Downgrade | Unexplained heterogeneity across studies |
| Indirectness | Downgrade | Differences in population, intervention, comparator, or outcome from the question of interest |
| Imprecision | Downgrade | Wide confidence intervals or few events |
| Publication bias | Downgrade | Suspected selective reporting of positive findings |
| Large effect | Upgrade | Very large or very consistent observed effects |
| Dose-response | Upgrade | Plausible dose-response gradient |
| Plausible confounding | Upgrade | Residual confounding would reduce rather than create the observed effect |
The body of evidence is then summarised in one of four certainty levels:
- High — we are very confident the true effect lies close to the estimate
- Moderate — the true effect is likely close to the estimate, but could be substantially different
- Low — the true effect may be substantially different from the estimate
- Very low — we have very little confidence in the estimate
GRADE certainty ratings are typically presented in a Summary of Findings table alongside the pooled effect estimates, allowing decision-makers to weigh the magnitude of the effect against the trustworthiness of the underlying evidence. GRADE is endorsed by Cochrane, the WHO, NICE (UK), and over 100 other organisations worldwide.
Reflection
Think of a public health question relevant to your interests. How would you specify the question for a systematic review? What databases would you search, and what inclusion/exclusion criteria would you set?
Minimum 20 characters required.
1. Which of the following is a limitation of narrative reviews?
2. What is the first step in conducting a systematic review?
3. Why should data extraction in a systematic review be carried out by two independent investigators?
Meta-Analysis: Data Types & Effect Models
Introduction and Overview
Section 1 covered how to identify and appraise the studies that go into a review. Section 2 turns to the quantitative half: combining the effect estimates from those studies into a single pooled estimate. The central design choice here — fixed-effects versus random-effects — is not arbitrary; it reflects an underlying assumption about whether all the studies are estimating the same true effect.
Learning Objectives
- Define meta-analysis and state its two principal objectives (pooled estimate; explore variation).
- Distinguish summary, group, and individual-patient data, and describe the tradeoffs of each.
- Compare the assumptions of fixed-effects and random-effects models and state when each is appropriate.
- Interpret pooled point estimates, confidence intervals, and study weights produced by either model.
16.3 What Is a Meta-Analysis?
A meta-analysis is “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” (Glass, 1976). It is a formal process for combining results from multiple studies and is considered the “gold standard” for providing summary information about health interventions.
Objectives of Meta-Analysis
The objectives are to: (1) provide an overall estimate of an association or effect based on data from multiple studies, and (2) explore reasons for variation in the observed effect across studies. Because it combines data from multiple studies, meta-analysis gains statistical power for detecting effects.
16.3.1 Types of Data in Meta-Analysis
Three types of data can be used in a meta-analysis, each with different capabilities:
| Data Type | Binary Outcome | Continuous Outcome |
|---|---|---|
| Summary estimate | Point estimate: RR, OR, RD, IR Precision: SE or CI | Point estimate: mean difference (MD) Precision: SE or CI |
| Group data | Cell values for treated and control groups (2×2 table) | Number, mean, and SD in each group |
| Individual patient data (IPD) | Raw data: outcome value (0 or 1) and individual characteristics | Raw data: outcome value and individual characteristics |
Summary data are most commonly used. Group data allow computation of various effect measures. IPD are the most flexible but rarely available—they allow evaluation of study-, group-, and individual-level variables as sources of heterogeneity.
16.4 Fixed- vs. Random-Effects Models
A fundamental decision in any meta-analysis is whether to use a fixed-effects or random-effects model:
Fixed-Effects Model
Assumes the true treatment effect is constant across all studies. Any variation among observed study results is due solely to within-study random variation (sampling error).
Where Ti is the observed effect from study i, θ is the true overall effect, and Vi = [SE(Ti)]² is the known within-study variance. Weights are computed as Wi = 1/Vi (inverse variance weighting).
Advantage: Does not require estimating between-study variance (τ²).
Limitation: The assumption of a constant effect across all studies is often untenable, and ignoring between-study variation can lead to Type I errors and confidence intervals that are too narrow.
Random-Effects Model
Assumes a distribution of true treatment effects across studies (heterogeneity), with additional variability beyond within-study sampling error.
Where ui is the random effect for study i, and τ² is the between-study variance (heterogeneity). Weights become Wi = 1/(Vi + τ²).
Result: Produces a similar point estimate to fixed-effects but with a wider confidence interval (because it accounts for between-study variation). Random-effects models are now more commonly used; the foundational estimator is from DerSimonian & Laird (1986).
Key Distinction
The fixed-effects model asks: “What is the single true effect?” The random-effects model asks: “What is the average of the distribution of true effects?” Random-effects models are generally preferred because the assumption of a constant treatment effect across all studies is rarely justified.
16.4.1 Weighting Methods
The most common weighting procedure is inverse variance weighting, applicable to both continuous and binary outcomes. For binary outcomes with sparse data, the Mantel-Haenszel procedure (Mantel & Haenszel, 1959) or the Peto method may be preferred. For continuous outcomes, when studies use different scales, standardised mean differences (effect sizes such as Cohen’s d or Hedges’ g) are used.
Reflection
Consider a meta-analysis of 10 studies examining the effect of a drug on blood pressure. Five studies were conducted in elderly populations and five in young adults. Would you expect a fixed-effects or random-effects model to be more appropriate? Why?
Minimum 20 characters required.
1. In a fixed-effects model, what is assumed about the true treatment effect?
2. What does τ² represent in a random-effects meta-analysis?
3. Which type of data provides the most flexibility for exploring sources of heterogeneity in a meta-analysis?
Forest Plots & Heterogeneity
Introduction and Overview
Section 2 produced a single pooled estimate. Section 3 introduces the visual and quantitative tools for inspecting the underlying variation that produced that estimate. The forest plot makes the constituent studies and the pooled result visible at a glance; heterogeneity statistics quantify how much the studies actually disagree with one another. Both are essential before trusting the pooled number.
Learning Objectives
- Read a forest plot: identify point estimates, confidence intervals, study weights, the null line, and the pooled diamond.
- Quantify heterogeneity using Cochran's Q, I-squared, and tau-squared, and interpret the conventional thresholds.
- Distinguish statistical heterogeneity from clinical and methodological heterogeneity.
- Decide when subgroup analysis or meta-regression is the appropriate response to detected heterogeneity.
16.5 Presentation of Results: The Forest Plot
The forest plot is the most important graphical output of a meta-analysis. It displays the point estimate and confidence interval of the effect observed in each study, along with the summary estimate.
Anatomy of a Forest Plot
Each horizontal line represents one study’s results. The length of the line is the 95% CI. The centre box marks the point estimate, and the area of the box is proportional to the study’s weight. The dashed vertical line shows the overall summary estimate. The diamond at the bottom represents the pooled estimate and its CI. The solid vertical line marks the null value (e.g., 0 for mean difference, 1 for ratio measures).
Watch a forest plot build itself one study at a time, see the PRISMA flow filter thousands of records down to twelve, and watch the pooled diamond emerge. Next ▶ advances scenes.
A 6-scene visualization of the systematic-review pipeline: PICO question, PRISMA filter funnel, forest plot construction one study at a time, heterogeneity check, pooled diamond, and the evidence pyramid.
If all study CIs overlap considerably and cluster near the summary estimate, there is little heterogeneity. If CIs are widely scattered and many do not overlap, heterogeneity is substantial. Studies may be ordered by publication year (to detect time trends), quality score, or effect size.
metafor package
The companion R script r-activities/HSCI_230_Lesson_2_Systematic_Reviews_and_Meta_Analysis.R runs an end-to-end meta-analysis — pooled estimate, heterogeneity diagnostics, and the figure for the paper — in three lines of metafor.
# install.packages("metafor")
library(metafor)
# Eight hypothetical RCTs of a smoking-cessation drug (events / total per arm)
dat <- data.frame(
study = paste("Trial", 1:8),
ai = c(35, 42, 28, 61, 19, 73, 31, 22), # events on drug
n1i = c(120,160,100,220, 85,240,130,90),
ci = c(22, 31, 19, 42, 15, 55, 21, 15),
n2i = c(120,160,100,220, 85,240,130,90)
)
# Compute log-OR per study with continuity correction
es <- escalc(measure = "OR",
ai = ai, n1i = n1i,
ci = ci, n2i = n2i, data = dat)
# Random-effects pooled OR (REML)
fit <- rma(yi, vi, data = es, method = "REML")
fit
# Forest plot
forest(fit, slab = es$study, transf = exp, refline = 1,
xlab = "Odds ratio (smoking cessation)")Reading the output. I² ~ 24% suggests low heterogeneity; the forest plot will show all CIs overlapping with the diamond. Try switching method = "FE" for a fixed-effect model and watch the diamond narrow. Heterogeneity diagnostics, publication-bias plots, and meta-regression all live in metafor.
R Reflect on what you just ran
Use the questions below to interpret the output you produced. Look at your console / forest plot before answering.
1. The pooled random-effects estimate from rma() was about OR = 1.59 (exp(0.46)). In plain language, what does that pooled odds ratio say about how the smoking-cessation drug compares to control across the 8 trials? Is the effect in a beneficial direction?
2. The output reported I² = 23.5% and Q-test p = 0.245. Using the benchmarks from the lesson (25% low, 50% moderate, 75% high), how would you describe the heterogeneity across these 8 trials, and does the Q-test agree?
3. Re-run the model with method = "FE" (fixed effects). Compare the width of the diamond on the forest plot to the random-effects version. Why does the fixed-effect CI become narrower, and when would that narrower CI be misleading?
16.6 Heterogeneity
Heterogeneity refers to variability among study results beyond what would be expected from random variation alone. It should always be evaluated in a meta-analysis.
16.6.1 Real vs. Artifactual Heterogeneity
An important distinction is between clinical heterogeneity (real differences between populations, interventions, and settings) and statistical heterogeneity (variation in observed results beyond chance). Clinical heterogeneity is always expected; the key question is whether statistical heterogeneity is also present.
16.6.2 Measuring Heterogeneity: Cochran’s Q and Higgins I²
Where wi are the study weights, Ti are the study effects, and θ is the pooled estimate. Under the null hypothesis of no heterogeneity, Q follows a χ² distribution with k−1 degrees of freedom. However, the Q test has low power when the number of studies is small, so a non-significant result does not rule out heterogeneity. Consider using a relaxed P-value threshold (e.g., 0.10 instead of 0.05).
I² (Higgins & Thompson, 2002) quantifies the proportion of variance between studies that is due to heterogeneity rather than chance. Benchmarks: 25% = low, 50% = moderate, 75% = high heterogeneity. An evaluation of possible causes should be undertaken whenever I² exceeds 25%.
16.6.3 Evaluating Causes of Heterogeneity
Identify a specific subgroup of studies defined by a characteristic of interest and examine the effect within that subgroup. However, results should be interpreted with caution—the best estimate for any subgroup is provided by considering all the evidence (Stein’s Paradox), not just the subgroup data. Subgroup analyses should be pre-specified in the review protocol.
Data are stratified by a factor thought to influence the treatment effect, and a separate meta-analysis is carried out in each stratum. The between-strata heterogeneity can be tested using QB = QT − ΣQS. A disadvantage is that individual strata may contain few studies.
A Galbraith plot plots the Z statistic (Ti/SE(Ti)) against the inverse of the SE (1/SE). The slope of the resulting line is the overall fixed-effect estimate, and lines at ±2 units from this line should encompass 95% of observations if there is no significant heterogeneity. Points outside these bounds are potential outliers contributing to heterogeneity.
Meta-regression is the most flexible approach: a weighted regression of observed treatment effects against study-level predictors (with inverse variance weights). It extends the random-effects model by adding predictors. Cautions: (1) even with RCTs, meta-regression is observational; (2) multiple comparisons inflate Type I error; (3) ecological fallacy applies since predictors are study-level averages.
Reflection
You conduct a meta-analysis of 20 studies and find I² = 82%. The forest plot shows widely scattered effect sizes. What steps would you take to investigate the causes of this high heterogeneity? Which methods from this section would you prioritise and why?
Minimum 20 characters required.
1. In a forest plot, what does the area of the box on each study line represent?
2. A meta-analysis reports I² = 75%. How should this be interpreted?
3. What is meta-regression used for in a meta-analysis?
Publication Bias, Influential Studies & Data Issues
Introduction and Overview
Sections 1–3 produced and inspected a pooled estimate. Section 4 addresses three threats that can survive correct technique: publication bias (some studies never make it to the literature), influential studies (a single trial driving the entire pooled result), and outcome-scale issues that make it inappropriate to pool studies that look superficially similar. Each of these is a place where a meta-analysis can produce a precise but misleading answer.
Learning Objectives
- Define publication bias and explain why it biases pooled estimates away from the null.
- Read a funnel plot, apply Begg's and Egger's tests, and interpret the trim-and-fill adjustment.
- Identify influential studies via leave-one-out sensitivity analysis and decide how to report their effect.
- Recognize when outcome-scale issues (binary vs continuous, time-to-event, dose-response) make pooling inappropriate.
16.7 Publication Bias
A critical concern in meta-analysis is publication bias—studies with statistically significant or favourable results are more likely to be published than those with null or unfavourable results. Consequently, published studies may represent a biased subset of all work conducted on a topic.
Why Publication Bias Matters
If the meta-analysis only includes published studies, and published studies tend to overestimate the effect, the summary estimate will be biased away from the null. This can lead to erroneous conclusions about the effectiveness of interventions.
16.7.1 Detecting Publication Bias: The Funnel Plot
A funnel plot displays each study’s SE (or its inverse, 1/SE) plotted against its estimated effect. In the absence of publication bias, the plot should resemble an inverted funnel—symmetric around the summary estimate, with small studies (large SEs) scattered widely at the bottom and large studies (small SEs) clustered near the top.
Asymmetry in the funnel plot suggests publication bias. For example, if studies with large effects and large SEs are present, but studies with small or null effects and large SEs are missing (a “gap” on one side), this suggests that null-result studies were not published. However, asymmetry can also arise from other factors, so interpretation should be cautious.
16.7.2 Statistical Tests for Publication Bias
Two commonly used tests evaluate the relationship between study results and their precision:
- Begg’s test: A rank correlation between effect estimates and their SEs. Simple but low power with few studies.
- Egger’s test: A linear regression approach that is generally more powerful at detecting publication bias (Egger et al., 1997).
Neither test is very sensitive when the number of studies is small (<20), and both may produce false positives when there are large treatment effects, few events per trial, or all trials are of similar size. For comprehensive guidance on interpreting funnel-plot asymmetry, see Sterne et al. (2011).
16.7.3 Trim-and-Fill Method
The trim-and-fill method (Duval and Tweedie, 2000) is a practical approach to assessing and adjusting for publication bias:
- “Trim”: Produce a funnel plot and sequentially omit the most extreme studies on one side until the plot is approximately symmetrical.
- Determine the centre of the trimmed, symmetrical plot (a new estimate of the treatment effect).
- “Fill”: Replace the omitted studies along with their hypothetical “counterparts” on the other side of the centre line.
- Redo the meta-analysis including both the original data and the hypothetical studies.
This provides an estimate of what the treatment effect would be if all studies had been published. The difference between the original and adjusted estimates indicates the potential impact of publication bias.
16.8 Influential Studies
It is important to determine whether individual studies have a profound influence on the summary estimate. A study might be much larger than others or have an extreme effect size. To evaluate this, sequentially delete each study from the meta-analysis and observe how the summary estimate changes.
In a meta-analysis of 25 studies with a pooled estimate of −2.121, one study (refid 218) was identified as a potential outlier in the Galbraith plot. Removing it changed the estimate to −2.011 (a 5% reduction in magnitude) and reduced I² from 95.6% to 88.1%. While the heterogeneity remained high, the analysis demonstrated that this single study had a meaningful influence on the results.
16.9 Outcome Scales and Data Issues
Published studies vary substantially in how they present data. Several practical issues arise:
Reflection
You produce a funnel plot for your meta-analysis and notice asymmetry—there appear to be “missing” studies with small effects and large standard errors. What are the possible explanations for this pattern beyond publication bias? How would you investigate further?
Minimum 20 characters required.
1. What pattern in a funnel plot suggests publication bias?
2. What is the purpose of the trim-and-fill method?
3. Why is sensitivity analysis (sequentially removing studies) important in meta-analysis?
Final Assessment
Bringing It All Together
Lesson 2 introduced evidence synthesis: the structured process by which the literature on a question is identified, appraised, and pooled into a single defensible answer. The arc moved from the seven steps of a systematic review, through the central design choice between fixed- and random-effects models, to the diagnostic tools (forest plots, heterogeneity statistics, funnel plots) that show whether the pooled answer should be trusted, and finally to the threats — publication bias, influential studies, outcome-scale issues — that survive correct technique.
The final assessment below asks you to integrate across all four sections: distinguishing systematic from narrative reviews, choosing between fixed- and random-effects, reading a forest plot, computing and interpreting heterogeneity, and recognizing when a precise pooled estimate is nonetheless misleading.
What you take away here connects directly to the capstone. A trustworthy mini-review for your study protocol is, in miniature, the same exercise: explicit inclusion criteria, structured search, transparent appraisal, and an honest accounting of what the existing literature can and cannot answer.
Key Takeaways from Lesson 2
- Systematic reviews are reproducible — they specify the question, search strategy, inclusion criteria, appraisal, and synthesis transparently, in contrast with narrative reviews that depend on the reviewer's preconceptions.
- Meta-analysis is the quantitative half: a pooled effect estimate, gained statistical power, and a structured way to explore variation across studies.
- Fixed- vs random-effects is a substantive choice — fixed-effects assumes one true effect, random-effects allows the true effect to vary across studies. Random-effects is the default when there is meaningful clinical or methodological heterogeneity.
- The forest plot makes the pooled story visible at a glance, and I-squared, Cochran's Q, and tau-squared quantify how much the studies actually disagree.
- Publication bias, influential studies, and outcome-scale mismatch can produce a precise but misleading pooled estimate — funnel plots, leave-one-out diagnostics, and pre-specified scale checks are essential safeguards.
- PRISMA is the reporting standard for systematic reviews and meta-analyses; trustworthy reviews follow it explicitly.
This assessment covers all sections of Lesson 2. You must answer all 15 questions correctly to complete the lesson. Review the feedback after each attempt.
Final Reflection
Reflect on the entire process of systematic review and meta-analysis. What do you see as the greatest strength of this approach compared to narrative reviews? What is the most challenging aspect, and how would you address it in your own research?
Minimum 20 characters required.
1. What is the primary purpose of a systematic review?
2. Which step of a systematic review involves specifying inclusion and exclusion criteria?
3. In inverse variance weighting, how are weights computed?
4. What key assumption distinguishes a random-effects model from a fixed-effects model?
5. In a forest plot, what does the diamond at the bottom represent?
6. Cochran’s Q statistic tests the null hypothesis that:
7. An I² value of 50% indicates:
8. Why might the Q test fail to detect heterogeneity even when it exists?
9. What does an asymmetric funnel plot typically suggest?
10. The trim-and-fill method is used to:
11. In a random-effects model, study weights are computed as:
12. Which effect size measure expresses the treatment effect relative to within-study variability?
13. Why is the choice of effect measure (RR vs. OR vs. RD) important in meta-analysis?
14. What is Stein’s Paradox in the context of subgroup analysis?
15. A sensitivity analysis reveals that removing one study changes the pooled estimate from RR = 0.65 to RR = 0.89. What should you conclude?