# Lesson 5: AI-Assisted Literature Searching and Reading *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a moment from the course's fictional Cedar Valley case. A scholarly search engine gives Jordan, the co-op student, a summary of the research, and it says most studies reported less loneliness. **Kiffer:** That was accurate. In fact, every sentence in the summary was right, as far as it went. **Sarah:** So that's a good summary, isn't it? **Kiffer:** Except it left out how weak the reviews judged the evidence to be, and that was the point the team most needed. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. This is Lesson five, AI-Assisted Literature Searching and Reading. Sarah, where do you want to begin? **Sarah:** Question one. I think this is how a lot of students feel. The tool gave me twenty papers. They're all on topic, and they're in good journals. What's missing? **Kiffer:** What's missing is everything below the cut-off. A scholarly AI search engine ranks its index by similarity of meaning and shows you the top of the list. It tells you nothing about how many relevant papers sit further down. So its recall, the share of all the relevant studies that a search finds, is limited by design, and you have no way to know how limited. **Sarah:** And the weakest case is a chat assistant answering from memory, with no search at all. **Kiffer:** Right. A 2024 study gave chat assistants the inclusion criteria of eleven published systematic reviews. Answering without a search, two versions of ChatGPT found about twelve and fourteen percent of the studies the reviews had included. Tools that search a scholarly index return real papers far more often, and none of them promises a complete set. **Sarah:** Can we put numbers on the difference between a good set and a complete one? **Kiffer:** Let's use a made-up example. You ask a scholarly AI search engine your question, and it shows you twenty papers. You read the abstracts and judge sixteen of them relevant. **Sarah:** So the precision, the share of what came back that's relevant, is sixteen out of twenty, or eighty percent. That's a good set. **Kiffer:** It is. Now suppose that a year later, a team publishes a systematic review on the same question. Their database searches find sixty-four relevant studies published before your search, and all sixteen of yours are among them. Treat those sixty-four as the full set. What was the recall of your top set? Take a few seconds. *(Pause)* **Sarah:** Sixteen divided by sixty-four is one quarter, so the recall was twenty-five percent. Three quarters of the relevant studies never appeared on my screen. **Kiffer:** And could you have calculated that on the day you searched? **Sarah:** No. I could work out the precision, because I read all twenty abstracts. The recall needs the total number of relevant studies, and I only learned that from the review a year later. **Kiffer:** That's what low and unknown recall means in practice. Wang and colleagues saw a similar pattern of high precision and lower recall when they asked ChatGPT to write Boolean queries for systematic review topics, and they suggested the approach might suit rapid reviews, where some loss of recall is acceptable. **Sarah:** What about a deep research agent? It runs many searches. Doesn't that close the gap? **Kiffer:** It casts a wider net, and it still reports a selection of what it found. However many searches produced the selection, you can't count from it. If a supervisor asks how many trials of link workers have been published, the answer has to come from a database search or a recent systematic review. **Sarah:** The other thing that surprises students is that the same question can bring back different papers in March and in June. Three things change underneath you. First, the model samples its words, so the same prompt can produce a different selection. Second, the index grows. Third, the model and the ranking method get updated, often without notice. How much does that matter? **Kiffer:** It depends on the task. An evidence scan accepts that it may miss studies, so you record the tool, the date and the prompt in your AI use log and state the limits. A systematic review promises a complete, documented search, so its core stays in bibliographic databases, and studies an AI tool finds are reported as identified by other methods. **Sarah:** I want to give these tools their due, though. The semantic matching is a real strength. **Kiffer:** It is. A query about community connectors can return papers about link workers when you've never heard that term. For someone new to a field, that's exactly what's needed, and a Boolean search finds those papers only if you know the synonym or the right subject heading. **Sarah:** Question two. Why does an AI-drafted search that runs without any error message still need testing? **Kiffer:** Because errors in a search strategy rarely show in its results. A wrong heading or a missing synonym gives you a list that looks plausible. In the course's fictional Cedar Valley case, the assistant drafted a PubMed search for Jordan, the co-op student, that included a subject heading called Link Workers. MeSH has no such heading. Sitting among other terms joined with OR, it added nothing, and the search ran normally. **Sarah:** And the draft also missed a real heading. **Kiffer:** Yes, the heading Social Prescribing, which the National Library of Medicine added to MeSH in 2025. The assistant's advice reflected the vocabulary from before its training cut-off. Those are the two typical failures on database details: a heading that fits the pattern and doesn't exist, and a fact that changed after training. Wang and colleagues found that in one of their experiments, fifty-five percent or more of the MeSH terms ChatGPT generated weren't in the vocabulary at all. **Sarah:** So checking every heading in the thesaurus catches those. What catches the rest? **Kiffer:** The known-item test. Here's a new example, and it's made up. A student asks an assistant for a PubMed search on whether heat warning systems reduce illness and death among older adults. The draft has three concept blocks joined with AND: heat warnings, health outcomes and older adults. Because the student mentioned older adults, the assistant also adds a last line with the NOT operator that drops any record whose title or abstract mentions a child, children or childhood. **Sarah:** That sounds sensible. The student doesn't want studies of children. **Kiffer:** Before searching, the student wrote down five papers that any good search on the question should find. The search returns three of the five. Both missing papers evaluate a city's heat warning system and report results for older adults. Which line would you suspect first, and why? Take a few seconds. *(Pause)* **Sarah:** I'd suspect the NOT line. The NOT operator throws out a record if the term appears anywhere in the fields it searches. An abstract that says young children and older adults are most at risk from heat would be removed, even though the paper reports on older adults. **Kiffer:** That's the right suspicion, and the age block was the other candidate, so you test them. Remove one line at a time and rerun the search. With the NOT line gone, all five known papers appear, and when you open the two missing abstracts, both name young children as another group at high risk. **Sarah:** So the fix is to delete the NOT line and remove studies of children at screening. **Kiffer:** Yes. When you screen titles and abstracts, you see each record before you remove it. A NOT line removes records you never see, including relevant ones. A related problem made Jordan's corrected search miss all three known reviews. There, requiring the loneliness and age concepts excluded reviews of social prescribing in general. **Sarah:** And then you take it to a librarian, with the question, the concept table, the full search with the AI-drafted lines marked and your changes noted, the record counts, and the known items with whether each was found. **Kiffer:** Yes. That lets the librarian spend the time on judgement, for example on whether you need a second database. **Sarah:** Question three. Gartlehner and colleagues found that a large language model extracted data from trial reports with about ninety-six percent accuracy. Students hear that number and ask why they should still check every value. **Kiffer:** Let's start with the arithmetic. Six errors in one hundred and sixty values is an error rate of 0.0375, about one value in twenty-seven. A comparison table of ten studies with ten fields holds a hundred values, so a similar error rate would leave about four wrong or missing values in it. **Sarah:** Here's my pushback. Suppose I check a sample. I pick ten values at random, and all ten are right. Haven't I shown the table is fine? **Kiffer:** Let's work it out. Suppose every value has the same chance of being wrong, 0.0375, independently of the others. So each value has a 0.9625 chance of being right. What's the chance that all ten values in your spot check come back correct? Take a few seconds. *(Pause)* **Sarah:** I multiply 0.9625 by itself ten times, which is 0.9625 to the power of ten. That comes to about 0.68. So about two times in three, my spot check would come back clean from a table that holds, on average, about four errors. A clean spot check of ten values does very little to tell that table apart from one with no errors. **Kiffer:** Right. The same reasoning answers a second question. What's the chance that all hundred values are right? That's 0.9625 to the power of one hundred, which is about 0.02. Only about two tables in a hundred would be free of errors. **Sarah:** And the trial reports in that study were the easy case. They were clearly reported and open access, with a fixed set of well-defined data elements. **Kiffer:** Right. The study can't tell us how a model performs on program evaluations or grey literature, which are harder to extract from. There's one more wrinkle. The calculation assumes every value is equally likely to be wrong, and real errors cluster: designs get upgraded, denominators get mixed up, qualifications get softened and items get missed. A random spot check spends most of its effort on fields that are usually right. **Sarah:** Which is why the supporting quotations matter. With a quotation for each value, checking all of them takes a few minutes per paper. **Kiffer:** And a quotation that doesn't contain its value is a warning sign in itself, like the sentence about the PRISMA guideline that was offered as evidence of a meta-analysis in Jordan's table. **Sarah:** Let me try one on denominators, since they're one of those clusters. This one's made up. A student asks for a one-sentence summary of a program evaluation, and the AI writes that seventy percent of older adults in the program became less lonely. The paper says one hundred and twenty older adults were referred, sixty completed the twelve-week follow-up, and forty-two of those had lower loneliness scores than at the start. There was no comparison group. **Kiffer:** Where does the seventy percent come from? **Sarah:** Forty-two divided by sixty is 0.7, so it's seventy percent of the people who completed follow-up. Out of everyone referred, it's forty-two out of one hundred and twenty, which is thirty-five percent. **Kiffer:** And what do we know about the sixty who didn't complete follow-up? **Sarah:** Nothing. If all sixty of them had lower scores too, it would be one hundred and two out of one hundred and twenty, or eighty-five percent. If none of them had, it stays at thirty-five percent. So for everyone referred, the data only tell us that the share with lower scores is somewhere between thirty-five and eighty-five percent. **Kiffer:** That's a wide range, and people who leave a program early often differ from those who stay. Is there anything else wrong with the sentence? **Sarah:** "Became less lonely" sounds as if the program caused it. With no comparison group, all we know is that scores were lower at twelve weeks than at the start. **Kiffer:** So a faithful sentence would read something like this. In one program evaluation with no comparison group, forty-two of the sixty participants who completed twelve-week follow-up, out of one hundred and twenty referred, had lower loneliness scores. It's longer, and it matches what the paper found. **Sarah:** And I can't fix this by telling the tool to be accurate? **Kiffer:** That's one of the surprising findings. Peters and Chin-Yee compared four thousand nine hundred summaries written by ten language models with the original texts. Model summaries were nearly five times as likely as expert-written summaries to contain broad generalizations, prompts asking for accuracy made the problem worse, and newer models tended to do worse than older ones. What helps is asking for specific things: the population, setting and design, results in the past tense, and the authors' certainty in their own words. Then you compare the summary with the authors' conclusion. **Sarah:** Question four. If a tool searched before it answered and every reference it gives is real, why check the citations? **Kiffer:** Retrieval changes which error you're most likely to meet. Without retrieval, fabrication is the big problem. Walters and Wilder found that fifty-five percent of the references from the older of the two ChatGPT versions available in 2023 were fabricated, compared with eighteen percent from the newer one. Newer versions have improved without solving it. A 2026 study by Bernstein and colleagues found that 7.1 percent of the references from a late-2025 version were fabricated, and only about half were correct in every detail checked. **Sarah:** And with retrieval? **Kiffer:** Fabrication becomes rare, and the main concern becomes a real source cited for a claim it doesn't make. Wu and colleagues checked the sources cited by seven language models across eight hundred medical questions. Between half and nine tenths of the responses weren't fully supported by the sources they cited. Even for a model with web search, about thirty percent of individual statements were unsupported. **Sarah:** So retrieval makes references real, and only reading the source shows whether one supports the claim. **Kiffer:** Let's try that on you, Sarah. Here's a made-up case, with invented numbers. An AI research report on heat and older adults says that telephone check-ins during heat waves reduced heat-related emergency visits among older adults by a third, and it cites a paper. **Sarah:** Okay. Step one, I search the exact title. **Kiffer:** You find it. The authors, year and journal match, and the DOI leads to the same record. It's a research article with no retraction notice, and the abstract is about heat and older adults. In the full text, you find a sentence saying that check-in calls during heat waves were followed by a one-third drop in heat-related emergency visits among older adults, with a numbered reference at the end of the sentence. How would you classify this citation? Take a few seconds. *(Pause)* **Sarah:** Step one passes, because the reference exists and the details match. Step two passes, because it's a research article on the topic. And for step three, I've found the claimed finding in the text, with the same one-third figure. So I'd call it accurate. **Kiffer:** What's the numbered reference doing at the end of that sentence? **Sarah:** Oh. That usually means the authors are describing someone else's study, probably in their introduction, which makes it a secondary citation. I matched the topic and the number and stopped. I need to find the original study they cited and check the claim there. **Kiffer:** Suppose you do. The original compares heat-related emergency visits among older adults enrolled in a check-in program in one city, over one summer, with the summer before. There's no comparison group. **Sarah:** Then the report's sentence overstates the original. It says the check-ins reduced visits, and a drop of a third compared with the previous summer could also come from a cooler summer. So what's the verdict now? **Kiffer:** It's inaccurate. Neither the paper it cites nor the original supports the sentence as the report worded it. If the original had supported it, the citation would be misattributed, and you'd only correct the reference. Here you fix the claim as well. Rewrite it to match the original study, something like this: in one city, heat-related emergency visits among older adults in a check-in program were about a third lower than the summer before, in a study without a comparison group. Then cite the original, and record the verdict and the secondary citation in the verification log. **Sarah:** What makes that one dangerous is that steps one and two passed cleanly. I only caught it because you asked about step three. **Kiffer:** That's why the steps are done in order, and why none is skipped when an earlier one passes. **Sarah:** Okay. Now the part where we see things differently. Should a student start a new topic with a deep research report? **Kiffer:** I think it's a good first step for orientation. In anything from a few minutes to half an hour, you get the vocabulary, the main debates and a list of candidate sources. Explaining a field and listing its terms are tasks of language, and that's where these tools are strongest. As long as you treat every claim in the report as a lead, the time saved is real. **Sarah:** My worry is what the first thing you read does to you. A fluent report sets the frame, and you read everything after it looking for what it told you. Think of the scholarly search engine summary in Jordan's case. It accurately said that most studies reported less loneliness, and it left out how weak the reviews judged the evidence to be, which was the point the team most needed. **Kiffer:** That's a fair concern, and that example is the strongest version of it. Every sentence in the summary was right, and the most important point was missing. **Sarah:** And the cost of checking is lopsided. Jordan's deep research report on wildfire smoke had thirty-four references, and checking just the first ten took a little over an hour. The one reference that didn't exist, invented for the lesson with obviously fictional authors and journal, was the only one of the ten with a statistic about older adults, the exact population Jordan asked about. **Kiffer:** I agree that checking is the expensive part. That same report also found genuine British Columbia studies of the 2003 fires and the BlueSky smoke forecasting system, and half of its first ten citations were accurate in every respect. Where I'd push back is on when you pay for the checking. Every claim you cite gets all four steps, and a source you only use to get oriented needs a lighter check. If a report gives you three good reviews and the right search terms, you've saved a lot of time, even if you set most of its prose aside. **Sarah:** I'd still open one recent review before reading any AI report, so that my first picture of the evidence comes from authors who rated its certainty. **Kiffer:** That's a reasonable habit, and it doesn't cost much. So where do we land? **Sarah:** I'm more cautious than Kiffer about the order, and we agree on the practical rule. Use a deep research report for vocabulary and leads, judge the strength of the evidence from a review you've opened yourself, including its certainty rating, and cite no source from the report until it has passed the four-step check. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, an AI search tool shows you a ranked top set. Its precision can be high, and its recall is low and unknown, so use it to get oriented and find a starting set, and use a database when a task needs completeness or a count. **Kiffer:** Second, test what an AI drafts against the system or the paper it describes. Check every heading in the thesaurus, run a known-item test, and check every extracted value in the paper, using its quotation to find the passage, because a clean spot check tells you very little. **Sarah:** And third, a real reference can still be the wrong citation. Steps one and two can pass cleanly, and only reading the source shows whether the finding is there, whether it's the paper's own and whether it matches the claim. **Kiffer:** If you'd like more practice, rework the spot-check problem from this episode with a different error rate, say one value in fifty, and work out how often a ten-value spot check would come back clean. **Sarah:** Next time, it's Lesson six, AI-Assisted Academic and Professional Writing, where Jordan turns the evidence scan into a briefing note and we look at feedback on drafts, authorship and disclosure. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson six.