# Lesson 9: AI in Public Health Practice *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** On the twenty-seventh of November 2002, a Chinese-language news report was published about a flu outbreak in mainland China. **Kiffer:** Canada's Global Public Health Intelligence Network, which the Public Health Agency of Canada now runs, received that report months before the outbreak reached Toronto. **Sarah:** It went on to the World Health Organization with an English header, and the full text was never translated. **Kiffer:** And Health Canada officials learned of the new disease, SARS, along with the rest of the world in February 2003. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. This is Lesson nine, AI in Public Health Practice, and we're starting with surveillance. **Sarah:** Question one. I'll begin with a claim from popular accounts at the time, that a Canadian company's software predicted the COVID-19 pandemic. Is that fair? **Kiffer:** The documented record is narrower, and it's still impressive. On the thirty-first of December 2019, BlueDot, a Toronto company, sent its clients an alert about a cluster of unusual pneumonia cases in Wuhan. Within two weeks, a team that included its founder, Kamran Khan, had published an analysis of air travel from Wuhan showing which cities travellers were most likely to fly to. That's an early flag and a useful estimate of where the infection might spread. **Sarah:** But BlueDot wasn't the only system that caught it. The same day, the World Health Organization's open-source intelligence platform picked up a report of the same cluster, and the World Health Organization's office in China saw a statement from the Wuhan Municipal Health Commission. **Kiffer:** Right. Several systems read the same open sources, so they found the same signal on the same day. Software made that speed possible, because it reads far more text than any team of analysts could. What happened after that depended on the people who received the signal. **Sarah:** Which is where the SARS story comes in. Canada's Global Public Health Intelligence Network, which the Public Health Agency of Canada now runs, received a Chinese-language news report about a flu outbreak in mainland China, published on the twenty-seventh of November 2002. That was months before the outbreak reached Toronto. The report went to the World Health Organization with an English header, but the full text was never translated, and Health Canada officials learned of the new disease along with the rest of the world in February 2003. **Kiffer:** So the software did its part, and the missing piece was people with the time and authority to translate the report and act on it. The COVID-19 record raised a related question. In March 2021 the Auditor General of Canada reported that the network hadn't issued an alert to give early warning when COVID-19 emerged, although it did email its daily report, with links to news articles, to subscribers including the provinces. **Sarah:** And an independent review made thirty-six recommendations about the network's role, its staffing and its technology. **Kiffer:** The value of any of these systems depends on the organization around it: who reads the output, who can raise an alert, and how the alert connects to a risk assessment. **Sarah:** There's a second place errors hide, and I find it easy to forget: the items a filter throws away. **Kiffer:** Here's a made-up example. A provincial team trials a vendor's news-monitoring service for one month. The service collects three thousand items about respiratory illness and marks one hundred and twenty of them as signals. Analysts read all one hundred and twenty and judge forty of them relevant. They also read a random sample of two hundred and forty of the two thousand eight hundred and eighty items the service discarded, and they find five that should have been flagged. What's the filter's estimated recall? Take a few seconds. *(Pause)* **Sarah:** Let me try it. Recall is the share of all the relevant items that the filter flagged. It flagged forty relevant items, and the analysts found five that it missed. So recall is forty divided by forty-five, which is about 0.89. That's a pretty good filter. **Kiffer:** How many of the discarded items did the analysts actually read? **Sarah:** Two hundred and forty, out of two thousand eight hundred and eighty. Oh, I see it. The five come only from the sample. The sample is one in twelve of the discarded items, so those five stand for about sixty relevant items in the whole discarded pile. **Kiffer:** So try the recall again. **Sarah:** Forty divided by forty plus sixty, which is forty out of one hundred, or 0.40. The filter found about two of every five relevant items and missed about three. **Kiffer:** That's the step that's easy to miss. The count from the sample has to be scaled up to the whole pile before it goes into the formula. And notice how your verdict changed. At 0.89 you were ready to trust the service, and at 0.40 the team would be missing three relevant items for every two it sees. **Sarah:** What about precision? Forty out of one hundred and twenty is one third, and that sounds poor as well. **Kiffer:** In surveillance I'd worry less about that. Eighty false signals a month cost analysts some reading time, while a missed signal can cost days of warning. It's the same trade-off that leads syndromic surveillance systems to tune their classifiers to catch most true cases at the cost of including some false ones. Recall is the figure I'd take back to the vendor. **Sarah:** Along with a caution about how firm it is. That estimate of sixty rests on five items. **Kiffer:** Right, so it's rough, and the team should keep sampling discarded items every month. And if the vendor's brochure says the service caught three outbreaks days before the official announcements, that describes three successes. It tells you nothing about recall, because recall depends on what the filter missed, and a list of successes never shows you that. **Sarah:** Question two. A vendor tells a health authority that its risk score has an AUC of 0.9. What does that tell us? **Kiffer:** It tells you how well the model ranks people. The AUC is the probability that a randomly chosen person who has the outcome gets a higher score than a randomly chosen person who doesn't. It doesn't depend on any threshold, and it doesn't tell you what share of the people you flag will have the outcome. Here's a made-up example that combines that point with external validation. It's the hardest one in this episode, so take it slowly. **Sarah:** Go ahead. **Kiffer:** A vendor's model predicts which older adults will have an emergency hospital admission in the next year. In the city where it was developed, one in five older adults had such an admission. At the default threshold, the model flags eighty percent of the people who go on to be admitted, and it also flags ten percent of the people who don't. Picture one thousand older adults in that city. Two hundred are admitted, and the model flags one hundred and sixty of them. Of the eight hundred who aren't admitted, it flags eighty. **Sarah:** So it flags two hundred and forty people, and one hundred and sixty of them are true cases. The positive predictive value is one hundred and sixty divided by two hundred and forty, which is two thirds. **Kiffer:** Right. Now a mostly rural health authority buys the model. There, only one in twenty older adults is admitted in a year. Assume the model flags people exactly as it did before, eighty percent of those who'll be admitted and ten percent of those who won't. What's the positive predictive value in the new region? Take a few seconds. *(Pause)* **Sarah:** Again, picture one thousand people. One in twenty is fifty admitted, and the model flags eighty percent of them, which is forty. Nine hundred and fifty aren't admitted, and ten percent of them is ninety-five. So it flags one hundred and thirty-five people, and forty of them are true cases. Forty divided by one hundred and thirty-five is about 0.30. **Kiffer:** That's right. In the first city, a care manager calls one and a half people for every person who goes on to be admitted. In the new region it's almost three and a half. And if the model scores people the same way it did before, the AUC hasn't moved at all, because it separates people with the outcome from people without it exactly as well as it did. The only change is that the outcome is rarer, and that alone cut the positive predictive value by more than half. **Sarah:** I'd guess the assumption that the model flags people exactly as before is generous. When a widely used proprietary sepsis model was validated independently at Michigan Medicine, its AUC was 0.63, against the 0.76 to 0.83 its developer had reported. **Kiffer:** Yes, and it missed about two thirds of the hospitalizations with sepsis at the threshold the authors evaluated. A model meets different patients, documentation and coding in a new setting, so its sensitivity can fall too, which would push the positive predictive value lower still. **Sarah:** And the one in twenty might itself be partly about distance. If people who live far from a hospital are admitted less often at the same level of illness, the label is measuring access as well as need. **Kiffer:** That's the proxy problem from the lesson. Obermeyer and colleagues found that a commercial algorithm that predicted health care costs ranked Black patients as lower risk than equally sick White patients, because less was spent on Black patients with the same level of need. They estimated that remedying the disparity would raise the share of Black patients receiving extra help from 17.7 to 46.5 percent. An admission label could under-rank rural residents in the same way, without location being one of the predictors. **Sarah:** There's a Canadian case that shows the careful version, though. St. Michael's Hospital in Toronto built CHARTwatch, its deterioration model, on its own data and evaluated it against control units. Non-palliative deaths on the general internal medicine unit fell from 2.1 percent to 1.6 percent. **Kiffer:** And the authors reported the uncertainty openly. The confidence interval reached no difference, and they cautioned about confounding. I'd point to its process as the pattern for public health teams to follow: local development, prospective evaluation and continued monitoring. **Sarah:** Question three. The study by Ayers and colleagues drew wide attention because evaluators rated a chatbot's answers as better and more empathetic than physicians'. They took one hundred and ninety-five questions that people had posted to a public forum on Reddit, each answered there by a verified physician, and gave the same questions to ChatGPT. Licensed health care professionals preferred the chatbot's response in 78.6 percent of evaluations. What has actually been shown? **Kiffer:** In the lesson's four levels of evidence, that's level two, where experts rate free-text answers. It's a real finding, and the design limits how far it reaches. **Sarah:** Because the physicians were volunteers answering strangers, in about fifty-two words on average against the chatbot's two hundred and eleven. Longer answers may have been rated as more empathetic partly because they were longer. And the evaluators couldn't check any answer against a real patient's situation. **Kiffer:** Right. What the study supports is narrower and still useful. These models can draft clear, warm replies to common health questions for a clinician to review. **Sarah:** The Goh trial is the one that puzzles people. The model working alone scored sixteen percentage points higher than physicians using conventional resources, yet physicians who had the model scored about the same as physicians who didn't. If the model does better alone, why not let it do the reasoning? **Kiffer:** The trial scored written reasoning about case vignettes with a rubric, which is a long way from a real patient with a complicated history, and it measured no patient outcomes. What it does show, at level three, is that handing people a capable tool doesn't automatically improve their work. How they divide the work and check the output matters, and that's why the B.C. framework makes organizations responsible for their staff's AI literacy. **Sarah:** Here's a made-up one for everyone. A health authority launches a public chatbot, and after six months it reports that eighty percent of users rated their conversation as helpful. Which level of evidence is that, and what does it leave unanswered? Take a few seconds. *(Pause)* **Kiffer:** It comes from a deployed system with real users, which is the setting of level four. What it measures is satisfaction. A resident who got a confident, wrong answer about eligibility might well rate it helpful, because they'd have no way to know it was wrong. To judge the service, you'd want a sample of conversations checked for accuracy against the approved documents, and some measure of whether people reached the right service. **Sarah:** Which brings us to the part where we disagree. Should a public health team put a generative chatbot in front of the public at all? **Kiffer:** I think there's a good case for it, with the right design. Some residents will put their question to a general-purpose chatbot anyway, after hours and in their own language. That kind of chatbot answers from general patterns. In the lesson's example from the fictional Cedar Valley Health Authority, a general-purpose chatbot got the program's age limit wrong, invented an online sign-up and promised the wrong meeting schedule. An agency's own chatbot, grounded in its approved documents, would answer from the program's actual rules. **Sarah:** My worry is what happens under the agency's name. The World Health Organization launched its health promotion chatbot as a prototype in April 2024, and within weeks journalists reported outdated answers, for example that an Alzheimer's drug approved in the United States in 2023 was still in trials. The disclaimer on its landing page didn't fix that. **Kiffer:** That chatbot was a generative design, answering from what its model learned in training. Grounding the answers in current documents addresses that particular failure. **Sarah:** It moves the risk somewhere else, though. A grounded chatbot repeats what the documents say, so an out-of-date fact sheet gets repeated faithfully, with the agency's authority behind it. The model can still misstate a passage. And the person typing might be an older adult in real distress after losing a spouse, who needs to reach a person. **Kiffer:** Those are real risks, and the federal guide lists deploying a chatbot for the public as a higher-risk use. I'd still build one, with a named owner for the documents, testing on real questions before launch, a clear statement that people are talking to AI, and routing of any sign of distress to a person. And I'd start with a staff-facing tool, like the referral helper in the lesson's worked demonstration. **Sarah:** I can agree with starting there. Clinic staff can spot a wrong answer about eligibility in a way a resident can't. **Kiffer:** Then here's a rule we can both work with. Start with staff. Ground the tool in approved, dated documents that a named person keeps current, test it on real questions, label it as AI, and review a sample of live conversations. Offer it to the public once that record shows it answers accurately and hands distress to people. **Sarah:** I'm still warier than Kiffer about the public step, and I'd follow that rule. **Kiffer:** That's a fair place to end up. Let's turn to everyday work. **Sarah:** Question four is about routine work, which is where most public health staff meet AI. One large study of that kind of use comes from the United Kingdom, where twenty thousand civil servants in twelve organizations used an AI assistant built into their office software for three months at the end of 2024. They reported saving an average of twenty-six minutes a day, with the largest savings in drafting. **Kiffer:** The savings were self-reported, and the report said human oversight was required at all times. So the picture is modest savings spread across many small tasks, which means the checks get repeated many times too. **Sarah:** Then my question is about one of those tasks, coding open-ended survey responses. How do you know the assistant is coding well enough? **Kiffer:** You code a sample yourself and measure agreement, one code at a time. Here's a made-up example. A health unit asks residents what got in the way of getting a flu shot. A staff member and an assistant each code the same forty responses, and the assistant is given the codebook and asked to quote the phrase that justifies each code it applies. **Sarah:** Give me the numbers for one code. **Kiffer:** For the code "language barriers," both apply it to two responses and both leave it off thirty-four. The staff member alone applies it to two more, and the assistant alone to two others. So they agree on thirty-six of forty, which is ninety percent. That's higher than the eighty-five percent that Jordan, the co-op student, and the assistant reached on unclear eligibility in the Cedar Valley case, where kappa was 0.69. So is the assistant coding language barriers well? Take a few seconds. *(Pause)* **Sarah:** I'll work out kappa. Each coder applied the code to four of the forty responses, which is 0.10. So the agreement expected by chance is 0.10 times 0.10, plus 0.90 times 0.90. That's 0.01 plus 0.81, or 0.82. Kappa is the observed agreement minus chance, so 0.90 minus 0.82, which is 0.08, divided by one minus 0.82, which is 0.18. That comes to about 0.44. **Kiffer:** So the answer is no. The raw agreement is higher than Jordan's, and the kappa is much lower. When a code is rare, most of the agreement comes from both coders leaving it off. Look only at the responses where either coder used the code. There are six, and they agreed on two of them. **Sarah:** And I'd guess the quoted phrases show why. If the assistant tags "I couldn't read the consent form" as a language barrier, that might be a language problem or a problem with the form itself. **Kiffer:** That's the kind of boundary a sharper definition settles. I'd add an example of each side of it, recheck agreement on a fresh sample, and once the assistant codes the full set, read every response it gives that code. Language barriers is exactly the sort of code an equity analysis depends on, so it's worth the extra hour even though it's rare. **Sarah:** And all of this assumes the responses went into an approved tool after identifying details were removed. **Kiffer:** Yes. Both the federal guide and the B.C. framework rule out putting personal information into tools that aren't authorized for it, and both make the person who uses the output accountable for it. **Sarah:** Let's pull it together with three things to take away. **Kiffer:** First, a filter's errors hide in what it discards. Estimate recall from a random sample of the discarded items, and scale that sample up to the whole pile before you use it. **Sarah:** Second, a model's reported performance belongs to the setting where it was measured. The AUC can stay the same while the positive predictive value falls by more than half, and the label itself can carry differences in access. So ask what the model predicts, where it was validated and what happens at the threshold. **Kiffer:** And third, match each claim to the evidence behind it. Expert ratings and user satisfaction are early steps toward knowing that a chatbot works, and for coding, agreement is measured one code at a time with kappa, because raw agreement can look high on a rare code even when the coders seldom agree about when to use it. **Sarah:** For more practice, rework the news-filter problem from this episode with a smaller sample of discarded items, say one hundred and twenty with two relevant, and see how much the recall estimate moves if a third relevant item turns up. **Kiffer:** Next time, it's Lesson ten, Responsible Adoption and Continued Learning, which brings the course together with equity and Indigenous data sovereignty, the wider governance of AI, the environmental cost of models, and a plan for keeping your skills current as the tools change. **Sarah:** Take care, everyone. **Kiffer:** See you in Lesson ten.