# Lesson 11: Evaluation and Careers in Health Promotion *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a program that sounds sensible. Take young people into prisons to deter them from crime. **Kiffer:** Those are Scared Straight programs. A Cochrane review of nine trials concluded that they increase delinquency compared with doing nothing at all. **Sarah:** So without those trials, they'd still look like a sensible idea. **Kiffer:** To most people, yes. So I'd want a funder to ask whether a tested program already fits, and Sarah has real reservations about that. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. We're on Lesson eleven, Evaluation and Careers in Health Promotion, and the first question comes up often when people read an evaluation report. **Sarah:** Question one. If a program doesn't work, does that mean its theory was wrong? I think the instinct is to say yes. The program was built on the Health Belief Model, nothing changed, so the Health Belief Model failed. **Kiffer:** That instinct skips several steps. A theory-based program is a chain of predictions. The activities are supposed to move certain constructs, and the constructs are supposed to move the behaviour. A program can come up empty because of a break anywhere in that chain, and only one kind of break tells you much about the theory. Let me give you a made-up example to work with. **Sarah:** Okay, go ahead. **Kiffer:** Picture a hypothetical program at a university, aimed at flu vaccination among students living in residence. Posters and short videos show how quickly flu spreads through a residence hall, which targets perceived susceptibility. They describe a week of missed classes, which targets perceived severity. Pop-up clinics in the residence lobbies lower a barrier, and text reminders act as cues to action. At the end of the flu season, the vaccination rate hasn't changed. **Sarah:** And now you're going to give me three versions of the follow-up data. **Kiffer:** Three versions. In the first, the process records show that the clinics ran on two of the ten planned days and half the reminders were never sent. In the second, everything was delivered as planned, but students' perceived susceptibility and severity didn't change from baseline. In the third, everything was delivered, perceived susceptibility and severity rose clearly from baseline, and vaccination still didn't change. **Sarah:** So here's the question for everyone listening. Which of the three versions tells you the most about the Health Belief Model itself? Take a few seconds. *(Pause)* **Kiffer:** The third one. In the first version, the program was barely delivered. That's a problem of fidelity and dosage, and the result says almost nothing about the theory, because students hardly received the program. Evaluators call this an implementation failure. **Sarah:** And the second version? The program ran as planned and the beliefs didn't move. **Kiffer:** That one tells you the activities didn't work on the constructs they were aimed at. Maybe the videos weren't persuasive, or students had already seen a hundred posters like them. That's a problem with the intervention's design. The theory predicts what happens after susceptibility and severity change, and they never changed, so that part of its prediction was never tested. **Sarah:** Which leaves the third version, where the beliefs moved and the behaviour didn't. I'd guess that's what evaluators call a theory failure. **Kiffer:** Yes, with two cautions. First, check what wasn't measured. If the clinics ran during class hours, a perceived barrier the survey never asked about could explain the result, and the Health Belief Model would have predicted exactly that. Second, one study is one data point. A well-evaluated program that disappoints counts as evidence about the theory in that population and setting, and it takes several results like it before anyone should conclude that the theory doesn't fit. **Sarah:** What strikes me is that you can only tell the three versions apart if you measured the beliefs at baseline and at follow-up. If you'd measured only vaccination, all three would look identical. **Kiffer:** That's the practical point. Theory tells you what to measure in the middle of the chain. Process data tell you whether the program reached people, and measures of the constructs tell you whether it moved what it was meant to move. With both, a disappointing result becomes something you can interpret and learn from. **Sarah:** Question two. Is attendance a result? I'd like to do this one myself. **Kiffer:** Then here's a made-up program for you. A community centre in Prince George runs strength and balance classes for adults over sixty-five, twice a week for forty-eight weeks, to prevent falls. Peer leaders who are older adults themselves co-lead the classes, which is modelling from Social Cognitive Theory, and the exercises get harder step by step so participants build confidence that they can move without falling, which is self-efficacy. Can you lay out the logic model? **Sarah:** Sure. Inputs are two kinesiologists, the peer leaders' time, the centre's space and the grant. The problem is falls among older adults, which lead to injuries and hospital stays. Outputs are the ninety-six classes and the eight peer leaders who were trained. Short-term outcomes are that one hundred twenty people attended at least half the classes, balance test scores improved, and fear of falling went down. And the long-term impacts are fewer falls and fewer hospital stays after falls. **Kiffer:** Most of that is right. Look again at the first thing you put under short-term outcomes, and ask whether it describes a change in the participants or something the program delivered. **Sarah:** Attendance. That's the program reaching people. Nobody's balance or confidence changed just because they walked in the door. So it's an output, it sits with the classes, and process evaluation checks it. It's also dosage, because it tells you how much of the program each person received. **Kiffer:** Right. It's an easy mistake to make, because attendance is the first number that comes in, and it feels like a result. A useful check for every box in the short-term column is to ask what's different now about the people, or the setting, the program set out to change. Balance scores and fear of falling pass that check. Attendance doesn't. **Sarah:** The attendance figure is still useful at the outcome stage, though. If balance improved most among the people who came to the most classes, that's a hint the program is doing the work. **Kiffer:** It's a hint, and a useful one, although people who come to every class may differ from the others in ways that matter, so it supports the case without settling it. Here's a question for everyone. The funder wants results at eighteen months. Which columns of Sarah's logic model can honestly report by then? Take a few seconds. *(Pause)* **Sarah:** The outputs can report from the first month, since they come from records. The short-term outcomes can report too, as long as balance and fear of falling were measured at baseline and then again at, say, six and twelve months. The long-term column can't, at least not honestly, and especially not the hospital stays. **Kiffer:** Can you show why with some numbers? **Sarah:** Let's make some up. Suppose six of the one hundred twenty participants would normally be hospitalized after a fall in a given year, which is five percent. If the program cut that by a third, you'd see four hospital stays where you'd otherwise see six. That's two fewer, and nobody could tell two fewer from chance. **Kiffer:** And that's before anyone asks whether hospital records can be linked to the participants. Impacts like that need a much larger group, years of follow-up, or both. So for a small program, the long-term column is usually a goal the program contributes to, and the evaluation reports on the outputs and the short-term outcomes. **Sarah:** One more thing to watch. If I read a report built on PRECEDE-PROCEED, the balance scores would be called impact, and the hospital stays would be called outcome. **Kiffer:** That's right. Green and Kreuter attach the two labels the other way around from the convention this course uses, while the idea of short-term and long-term change is the same in both. Check a report's definitions before you compare it with another one. **Sarah:** Question three. If the numbers improve after a program, how much of the change belongs to the program? **Kiffer:** A before-and-after change tells you that something changed. On its own it can't tell you why. The confounds from the lesson, history, maturation, testing, regression to the mean, selection bias, attrition and diffusion of treatment, are a list of other reasons the numbers might move. A comparison group is the main tool for dealing with several of them. **Sarah:** Is there a real example of a comparison doing that work? **Kiffer:** A well-known one from Vancouver. Insite, the first legal supervised injection site in North America, opened in 2003. In 2011, researchers published a study in The Lancet comparing overdose death rates before and after it opened. Within five hundred metres of Insite, the rate fell by thirty-five percent. In the rest of the city, over the same period, it fell by 9.3 percent. **Sarah:** So the rest of the city is the comparison. It shows roughly how much the death rate would have changed anyway, from everything else going on in Vancouver in those years. **Kiffer:** That's the logic. Anything that affected the whole city would show up in both areas, and the gap between them is the part that's hard to explain without Insite. **Sarah:** Can we try a made-up case where that logic runs into trouble? That seems like the harder version of this question. **Kiffer:** Picture a hypothetical peer-led vaping prevention program, built on Social Cognitive Theory, in one school district in the BC Interior, with a similar district nearby as the comparison. Both districts survey students aged fifteen to seventeen in the spring of 2019 and again in the spring of 2021. In the program district, thirty percent of students report vaping in the past month at baseline, and twenty percent at follow-up. In the comparison district, it goes from twenty-eight percent to twenty-four percent. **Sarah:** So here's the question. The program district dropped ten percentage points. How much of that drop would you credit to the program? Take a few seconds. *(Pause)* **Sarah:** I'd start with the comparison district. It dropped four points without the program, so four points is my estimate of what would have happened anyway. Ten minus four leaves six points that I'd credit to the program. **Kiffer:** That's the right first step, and it matters a lot here because of what happened in between. In 2020, British Columbia raised the tax on vaping products from seven to twenty percent, capped nicotine concentration, and restricted flavoured products to age-restricted stores. The COVID-19 pandemic also led the province to suspend in-class learning that spring. Those events hit both districts, so as long as they hit them about equally, the comparison absorbs them. Without it, you might have credited the program with all ten points. **Sarah:** So the six points is the program's effect. **Kiffer:** Here's one more invented detail. In 2020, the city in the program district also passed a bylaw keeping vape shops away from schools. The city in the comparison district didn't. Now what's in the six points? **Sarah:** The program and the bylaw, together. The comparison only removes events that hit both districts. Anything that happened in only one of them stays mixed in with the program's effect, and this design has no way to pull them apart. It's the history confound again, at the local level. **Kiffer:** That's the trap in this example. The quasi-experimental design asks whether anything else happened in the intervention community, and here something did. So the honest claim is that vaping in the program district fell six points more than in the comparison district, over a period when the program district also had a new bylaw. **Sarah:** Is there anything that could help separate the two after the fact? **Kiffer:** A few things. Process data, if some schools got the full program and others got only part of it. They all lived under the same bylaw, so a bigger drop in the full-dose schools points toward the program. The theory helps too. If refusal self-efficacy rose among participants, the program moved the construct it was built to move, and a bylaw about shops is unlikely to do that. Interviews with students could add why they stopped or never started. Each piece is incomplete on its own, and together they make a stronger case. **Sarah:** Which is the lawyer comparison from the lesson. You build the case from several kinds of evidence. **Kiffer:** Question four. Where does theory actually show up in a health promotion job? Students sometimes hear the careers material as a separate topic from evaluation and theory. **Sarah:** And I think the answer is that in most settings, theory shows up on paper first. **Kiffer:** Yes. A government agency builds a funding announcement on a framework it has adopted. A community organization answers with a grant application that justifies its plan with theory. A consultant writes the logic model and the evaluation plan. A review panel of experts and agency staff reads all of it and decides whether the plan makes sense, and a program officer oversees the work once it's funded. So let's read a made-up application the way a reviewer would. **Sarah:** Here it is. A hypothetical nonprofit in Kamloops asks for one year of funding. Its plan says, we will run twelve after-school workshops on stress for youth aged fourteen to eighteen. We expect the mental health of young people in Kamloops to improve. We will evaluate the program with a satisfaction survey at the end of each workshop. **Kiffer:** Here's the question for everyone. The evaluation plan is a satisfaction survey at the end of each workshop. What can that survey never tell the funder? Take a few seconds. *(Pause)* **Sarah:** Satisfaction is generally treated as process data, part of how participants received the program. It can't tell the funder whether anything changed in the young people, because there's no baseline, and it doesn't measure anything a theory says should change. **Kiffer:** And that points to the deeper gap. The plan names no theory, so nothing in it says what ought to change between the workshops and better mental health. How would you fix it? **Sarah:** I'd pick a theory that fits what the workshops do. Say older peers model coping skills and students rehearse them. That's Social Cognitive Theory, so the short-term outcomes are coping self-efficacy and intentions to seek help, measured at baseline and again a few months after the workshops end. The process evaluation counts the sessions delivered and how many each student attended, and checks whether each session followed the plan. **Kiffer:** And the sentence about the mental health of young people in Kamloops? **Sarah:** That's an impact, and twelve workshops in one year can't move a whole city's youth mental health in a way anyone could measure. I'd keep it as the long-term goal the program contributes to, and promise outcomes the program can show within the grant. **Kiffer:** That rewrite is the kind of plan a reviewer can work with. It also shows how the parts of an application hang together. The problem statement is the problem column of the logic model, the theory-based plan says what ought to change, and the evaluation plan follows from that. The budget and timeline become the yardstick for process evaluation. Every career setting in the lesson, from a public health unit to a consulting firm to the First Nations Health Authority, asks for that same combination. **Sarah:** Okay. Now the part where we disagree. Funders increasingly ask organizations to use a model program from a registry, or to justify why none fits. I have real reservations about that. **Kiffer:** Then make the case, and I'll tell you where I agree with it. **Sarah:** The programs in registries are the ones somebody could afford to test, often several times and often in the United States. A program developed by a small community organization, or an Indigenous-led program built on a community's own knowledge, may work just as well and never have been evaluated in a way a registry scores. So a model-program requirement tilts funding toward what's already been studied, and the work of justifying an exception falls hardest on the smallest organizations. **Kiffer:** That's a fair concern, and registries do reflect who had research funding. Where I'd push back is on what the requirement protects people from. Programs that sound sensible can do harm. A Cochrane review of nine trials of Scared Straight programs, which take young people into prisons to deter them from crime, concluded that these programs increase delinquency compared with doing nothing at all. Without those trials, the programs would still look like a sensible idea to most people. **Sarah:** That's a strong example. But it's an argument for evaluating programs, and community-developed programs can be evaluated too. In First Nations communities, the principles of ownership, control, access and possession, known as OCAP, mean the community controls its own data and how it's used. A community that sets its own goals and decides how to measure them, which is empowerment evaluation from the lesson, is producing evidence as well. **Kiffer:** I agree that community-defined evaluation produces evidence, and I'd like registries to find better ways to include it. I'd still want a funder to ask whether a tested program already fits, because starting from scratch every time throws away what earlier evaluations learned. **Sarah:** And I'd want the funder to pay for the evaluation when a community builds its own program, so the cost doesn't land on the organization with the least capacity. **Kiffer:** I can agree with that. So here's a rule we can both accept. Where a tested program fits the community, start there and record every adaptation in the process evaluation. Where none fits, build the evaluation in from the first day, with the community defining what success means, so the program can join the evidence base on its own terms. **Sarah:** I'm more skeptical of registries than Kiffer is, but that's a rule I'd follow. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, a disappointing result has more than one explanation. Measure delivery, and measure the constructs the theory targets, so you can tell an implementation failure from a theory failure. **Kiffer:** Second, keep the columns of the logic model straight. Attendance is an output. Short-term outcomes are changes in the people or settings a program targets, which it can measure within its own lifetime, and impacts on the health problem usually take years and larger numbers. **Sarah:** And third, a before-and-after change shows that something changed. A comparison group removes the effect of events that hit both groups about equally, and anything that hits only one group stays mixed in with the program's effect. **Kiffer:** If you'd like more practice, go back to the vaping example from this episode. Change one detail at a time, such as a bylaw in the comparison city, or a program district chosen because its vaping rate had just spiked, and work out what the six-point difference can and can't tell you. **Sarah:** This is the last episode of Office Hours for the course, since Lesson eleven is the final lesson. Thanks for working through all eleven lessons with us. **Kiffer:** Take care, everyone. **Sarah:** Thanks for listening, and all the best in the work ahead.