# Lesson 8: AI for Data Analysis and Coding in R *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** In October 2020, Public Health England announced that fifteen thousand eight hundred and forty-one positive COVID-19 cases had been left out of the reported daily case figures. **Kiffer:** Those cases were from the twenty-fifth of September to the second of October. Some files of positive results were larger than the maximum file size of the process that loads them into central systems. **Sarah:** So the results arrived, and part of them never made it into the totals. **Kiffer:** People still received their own results, and the outstanding cases were passed to contact tracing once the problem was found. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. This is Lesson eight, AI for Data Analysis and Coding in R, and Sarah's starting with errors that make no noise at all. **Sarah:** Question one. In the lesson, Jordan, the co-op student with the fictional Cedar Valley Health Authority, was given two pieces of AI-written code that ran without any message and still gave wrong answers. I can accept that, but it still feels strange to me. If R is strict enough to stop over a misspelled column name, why doesn't it stop when a missing-value code gets added into a score? **Kiffer:** Because from R's point of view, nothing went wrong. A misspelled name asks for something that doesn't exist, so R has nothing to work with and stops. Converting a factor to numbers is a perfectly legal request, even though R's own help page for factors warns that applying as.numeric to a factor is meaningless. What comes back is each answer's position in the list of levels, and in Jordan's data the fourth position was the missing-value code. **Sarah:** So the code did exactly what it was told, and what it was told didn't match what the data mean. **Kiffer:** That's the gap where silent errors live. The age groups were the same story. The cut function includes the upper end of each interval by default. That's a legitimate convention, and it happened to disagree with the labels Jordan wanted. In both cases the code rested on something nobody had checked, the order of the levels or a default setting, and code like that runs without complaint. **Sarah:** Does this kind of error happen outside teaching examples? **Kiffer:** It does, including in public health reporting. In October 2020, Public Health England announced that fifteen thousand eight hundred and forty-one positive COVID-19 cases, from the twenty-fifth of September to the second of October, had been left out of the reported daily case figures. Its statement said that some files of positive results were larger than the maximum file size of the process that loads them into central systems. People still received their own results, and the outstanding cases were passed to contact tracing once the problem was found. **Sarah:** So records went in at one end and were missing from the totals at the other. And that's the job of the plainest check we have, counting rows before and after every step. **Kiffer:** Right. Here's a hypothetical example that combines that idea with missing values, built on the real survey numbers. Suppose Jordan later asks the assistant for the percentage scoring six or higher in each age group, all in one step. The code counts everyone in a group who scored six or higher and divides by the number of people in the group. For the oldest group it reports 46.1 percent, where the lesson's table said 51.7. The oldest group has four hundred and eighty-six respondents, and four hundred and thirty-three of them have a score. What went wrong, and which figure answers Jordan's question? Take a few seconds. *(Pause)* **Sarah:** Working back from the lesson's figure, 51.7 percent of four hundred and thirty-three is about two hundred and twenty-four people who scored six or higher. Divided by four hundred and eighty-six, that same count gives the 46.1 percent. The fifty-three people without a score are sitting in the denominator, so the code treats them as if they had screened negative. Their answers are unknown, and the code read unknown as no. So 51.7 is the figure Jordan wants, the share among people who have a score. **Kiffer:** Exactly. And which check would catch it? **Sarah:** Adding up the number of respondents across the four groups. If it comes to four thousand and forty-five, the people without a score are still in. It should come to three thousand five hundred and eighty-three. The survey's own yes-or-no version of the cut-off would also give 51.7 as a second method. **Kiffer:** There's one more layer that makes this worse than a small shift. The share without a score differs by age group. It's about eleven percent in the oldest group and about fourteen percent at ages forty-five to sixty-four. So the error pulls the forty-five to sixty-four group down by eight points, from 57.9 to 49.9, and the oldest group down by about five and a half points. The gap between those two groups shrinks from about six points to about four. **Sarah:** So a denominator error can change a comparison as well as a level, and every number in the wrong table would still look reasonable. **Kiffer:** That's why the rule in Jordan's plan, that respondents without a complete score are excluded and counted, earns its place. Once it's written down, you know what the group sizes should add up to before you see them. **Sarah:** Question two. What makes a check a real check? I'll admit that when I think of a second method, my first instinct is to ask the assistant to do it a second way. **Kiffer:** That instinct is worth testing. Go back to the scoring step, before the error was found. Suppose Jordan asks the assistant to double-check its scoring code. The assistant writes a second version that converts each item to a number in the same way and adds them with a different function, one called rowSums, which adds across columns. It reports that the two versions agree for all four thousand and forty-five respondents. Is the score confirmed? Take a few seconds. *(Pause)* **Sarah:** No. Both versions turn each answer into a number the same way, by the position of its level, so both give the missing-value code a four. They had to agree. Two methods that share a mistake can't catch it. **Kiffer:** Right. On the real survey file, the two versions do agree exactly. And with the wrong score, the percentage for the oldest group comes out at 52.3 percent. The correct figure is 51.7, and a difference that small is easy to miss by eye. **Sarah:** So what makes a second method independent? **Kiffer:** It reaches the answer by a route that doesn't share the assumption you're worried about. The survey team computed its own score separately, so it's independent of Jordan's code. The survey's yes-or-no variable for the cut-off is another. Asking the assistant whether its code is correct fails the same test, because it checks its reasoning with the same reasoning. A more useful question is what the code assumes about the data, and what could make it give a wrong answer without an error. **Sarah:** And when there's no independent number at all, which is the situation with the Connector program's own screening data, the fallback is a known-answer test. Let me try designing one for the screen-positive rule, a score of six or higher. I'd invent a few respondents at the two ends of the scale, someone with a three and someone with a nine, and check that the three comes back as no and the nine as yes. **Kiffer:** Suppose the assistant had written the rule as greater than six. What would your test show? **Sarah:** The three would still come back no and the nine would still come back yes. So the test passes and the rule is wrong. I picked the easy cases. Everyone with exactly six would drop out, and my test never asks about a six. I need rows right at the threshold, a five, a six and a seven, plus one respondent with the missing-value code, who should come back missing. **Kiffer:** That's the test. And the stakes are real. On the survey data, the greater-than-six version puts the oldest group at thirty-three percent, against the correct 51.7, because the eighty-one older respondents who scored exactly six fall out of the positive group. **Sarah:** It's the same lesson as the age groups. Errors gather at the edges, so that's where the test rows go. **Kiffer:** Yes. And how much of this you do should match the stakes. A figure going to a planning meeting deserves every check. A quick look at one variable might need only a count. **Sarah:** Question three. How many analyses are hiding inside one? This is the garden of forking paths, and I think it's one of the hardest ideas in the lesson, because it says you can have a false-positive problem with a single analysis. **Kiffer:** Let's build up to it with numbers, using a hypothetical analyst working with the same survey. This analyst hasn't written a plan, and makes three yes-or-no choices after looking at the data. A threshold of six or seven. The youngest group starting at sixteen or at eighteen. The oldest group starting at sixty-five or at seventy. Suppose, just for this example, that age has no real link with loneliness. How many versions of the analysis are there, and roughly what's the chance that at least one of them shows a difference at the 0.05 level, if each version runs one test and we treat them as independent? Take a few seconds. *(Pause)* **Sarah:** Three choices with two options each is two times two times two, so eight versions. The tempting answer is eight times five percent, which is forty percent. That reasoning would give a hundred percent for twenty versions, though, and the lesson's figure for twenty is about sixty-four percent. So I'll work it from the other side. Each version has a 0.95 chance of not producing a false positive. For all eight to come up clean, that's 0.95 to the power of eight, which is about 0.66. One minus 0.66 is about 0.34, so roughly a one in three chance. **Kiffer:** That's right. Now the harder layer. The question is about older adults, so suppose that in each version the analyst compares the oldest group with each of the three younger groups. How many comparisons is that? **Sarah:** Eight versions times three comparisons, so twenty-four. **Kiffer:** And 0.95 to the power of twenty-four is about 0.29, so the chance of at least one false positive is about seventy-one percent. **Sarah:** That's a striking number, but I'd push back on it. These comparisons aren't independent. They use the same respondents, and a threshold of six and a threshold of seven will mostly pick out the same people. **Kiffer:** That's the right critique. Correlation pulls the real figure down, sometimes a long way. The direction stays the same, though. More paths give noise more chances to look like a finding. **Sarah:** And the forking-paths point is that the analyst never actually runs all twenty-four. **Kiffer:** Exactly. The analyst looks at the data, decides that seventy seems a natural place to start the oldest group, and runs one analysis. It feels like one test. With different data, a different path would have looked natural, so the error rate belongs to the whole set of paths. Gelman and Loken stressed that this can happen through ordinary, honest judgement. **Sarah:** Analysts made choices like these long before assistants existed. Where does AI come in? **Kiffer:** It changes the cost. Each new version used to mean writing more code by hand, and now it takes a sentence, and assistants tend to offer more analyses than you asked for. A dated plan fixes the threshold, the groups and the tests before any results exist. You can paste the plan into the prompt, and once the script is finished you can ask the assistant to list every place where the script departs from the plan. **Sarah:** Question four. Where can your data go? The rules make sense to me, but I think students tend to assume the data dictionary approach is always safe, and it's worth testing that. **Kiffer:** Let's do it with a made-up prompt. Suppose Jordan moves on to the Connector program's own participant records, which hold names, dates of birth, clinic, connector notes and loneliness scores at referral and follow-up. Jordan writes a careful prompt for the team's approved enterprise assistant, with the dictionary entries, the question and a request for code in small steps. At the end they add, here are the first three rows so you can see the format. **Sarah:** That last line is the problem. Those three rows are personal health information, and the pilot's rules say that never goes into the assistant, even the approved one. Removing the names wouldn't fix it. A date of birth and a clinic in a small rural community can point to one person. **Kiffer:** And the request behind that line is reasonable. Seeing the format does help the assistant. So the fix gives it the format without the people. That's a synthetic stand-in, a few invented rows with the real column names and the real codes, including the missing-value code, and the code then runs on the real records on the health authority's own network. **Sarah:** Could the dictionary itself be a problem? **Kiffer:** It can. Some dictionaries list how many records fall in each category. Names and codes are usually fine to share. A line saying that one clinic has only two participants is a count for a small category, and in a small rural community that can point to particular people. I'd strip the counts before a dictionary goes anywhere. **Sarah:** Then why not go further and generate realistic synthetic data from the real records? That would give the assistant something closer to the real thing. **Kiffer:** Realistic synthetic data, made with tools such as the R package synthpop, are derived from the real records, so they can carry some disclosure risk, and the data steward has to approve them. The random stand-in is safe because it copies the structure and nothing else. For getting code to run, the structure is usually enough. **Sarah:** Here's one I think is genuinely hard. Suppose some of the Connector records come from a First Nation, shared under a data sharing agreement that doesn't mention AI. Jordan plans to use only the dictionary and a synthetic stand-in for those records, so no record ever leaves the secure network. Is that allowed under the OCAP principles? Take a few seconds. *(Pause)* **Kiffer:** Not yet. Under the OCAP principles, Control means the community controls the research and information processes that affect it, and that includes the choice of tools. The safe default is to treat any AI use the agreement doesn't mention as unapproved until the community's data governance body has decided, even when the tool receives only a description of the data. **Sarah:** So the privacy rules ask whether a record moves, and OCAP also asks who gets to decide. **Kiffer:** Yes, and the request goes through the channels the agreement names. An agreement can also be written to address AI directly, saying which tools may be used, with what kinds of data, and how the community will see the code and the results. **Sarah:** Okay. Now the part where we disagree. I'm not convinced a beginner can learn to read R well without writing much of it. Reading a line and nodding along feels like understanding. The developers in the METR trial felt faster while they were measurably slower, and I worry that students can feel they understand code they couldn't have produced or fixed. **Kiffer:** That's a fair worry, and the METR point about impressions does carry over. The trial itself measured speed for experienced developers working on their own large projects, and METR said plainly that it doesn't show most programming work is slowed by AI. For a public health student, the realistic alternative to an assistant is often no analysis at all. I think the fluency that checking needs is reading code, recognizing common functions, understanding data types and interpreting output, and asking for line-by-line explanations of code you're actually running builds it. **Sarah:** But checking is exactly where I'd want the writing practice. Jordan's errors were found by checks that someone designed, like the range check and the youngest and oldest age in each group. Designing a good check means imagining how the code could fail, and I suspect you learn that partly by making those mistakes yourself. **Kiffer:** I agree that designing checks is the skill that matters most. Where I'd differ is on how you get it. Finding an assistant's error with factors or boundaries can teach much of what making that error yourself would teach. And a student who completes more real analyses with an assistant is likely to meet more kinds of errors. **Sarah:** Only if the student reads the output every time. A grey suggestion accepted with one key is the risk I keep coming back to. **Kiffer:** That's a real risk, and it's why I'd start beginners in a chat assistant, where they run every block themselves. So where does that leave us? **Sarah:** I'd still want students to write small pieces themselves, and I'm more cautious than Kiffer here. But we agree on a practical rule. Before you run any block, predict what it will print. Before a block goes into your script, be able to explain each line in your own words. And every so often, make a small change yourself, like a new threshold, before you ask the assistant. **Kiffer:** I'm happy with that rule. Predicting the output is the part I'd hold onto most, because it shows you straight away whether your reading of the code was right. Let's pull it together with three things to take away. **Sarah:** First, silent errors gather where code meets the meaning of the data, at missing-value codes, denominators and boundaries. Count rows and missing values at every step, and check that group sizes add up to what your plan says they should. **Kiffer:** Second, a second method counts only when it reaches the answer by a different route. Two versions that share an assumption can't test it, and known-answer tests belong right at the boundaries. **Sarah:** And third, decide the analysis before you see the results, and decide where the data may go before you open any tool. A dated plan limits the forking paths, and the classification of the data, together with the community's decision for First Nations data, settles what an assistant may receive. **Kiffer:** If you'd like more practice, add a fourth yes-or-no choice to the forking-paths problem from this episode, work out the new chance of at least one false positive, and then say which line of a dated plan would have removed each choice. **Sarah:** Next time, it's Lesson nine, AI in Public Health Practice, where we move from a single analysis to the uses of AI across public health, from surveillance to health communication. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson nine.