# Lesson 4: AI for Learning and Study *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** In a 2006 study of retrieval practice, some students kept rereading a passage, and others were tested on it. **Kiffer:** The students who kept rereading predicted they'd remember it better than the students who were tested. A week later, they remembered less. **Sarah:** So the students who felt surer were the ones who remembered less. **Kiffer:** And a clear, fluent explanation from an assistant is a very easy way to get that sense of being sure, whether or not anything has stuck. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. This is Lesson four, AI for Learning and Study, and we're starting with practice sessions that seem to go perfectly. **Sarah:** I think this one catches a lot of people. You sit down with an assistant, you work through twenty problems, every answer is correct, and it feels good. Why can't you count that as learning? **Kiffer:** Because the session measured what you could do with help, and what you care about is what you can do alone. The clearest evidence is the 2025 experiment by Bastani and colleagues, with nearly a thousand high school students in Turkey. Students who practised with a general assistant scored forty-eight percent higher than the control group on the practice problems, and seventeen percent lower on a later test taken without it. **Sarah:** And what worries me most is that they didn't notice. **Kiffer:** Right. The students who'd used the general assistant didn't perceive that they'd learned less. The students who'd used the tutoring version believed they'd done significantly better than the control group, and their test scores were about the same as the control group's. **Sarah:** That fits a much older finding. In the 2006 study of retrieval practice, students who kept rereading a passage predicted they'd remember it better than students who were tested on it, and a week later they remembered less. So the feeling of learning and the learning itself can come apart, and a clear, fluent explanation from an assistant is a very easy way to get the feeling. **Kiffer:** That's why it's worth using a test that doesn't depend on the feeling. One of the simplest is calibration. Before a self-test, predict your score, and afterwards compare. Sarah, do you want to set up an example? **Sarah:** Sure, and it's made up. Imagine a student studying a lesson with four sections and a four-question knowledge check at the end of each one. For sections one, two and three, they practised with a general assistant and mostly asked it for answers. For section four, they quizzed themselves with their notes closed. Before each check, they predicted their score. Their predictions were four, four, four and three. Their actual scores were two, three, two and three. **Kiffer:** So here's the question for everyone listening. How far off was this student's judgement overall, and where does the gap come from? Take a few seconds. *(Pause)* **Sarah:** Overall they predicted fifteen out of sixteen and scored ten, so they overestimated by five questions. And all five are in the first three sections, where they predicted twelve and scored seven. In section four they predicted three and got three. **Kiffer:** That's the pattern the Bastani result would lead you to expect. The sections practised with answers on hand felt mastered, and the checks showed they weren't. I'd still be careful about how much one student's sixteen questions can show. Section four might have been easier, or the student might simply have had more practice by then. So I'd treat it as a reason to change something and see whether the gap closes. **Sarah:** And the change would be to move one step to the unaided side. Practise the next section with a tutor set to withhold answers, or with the assistant closed, and see whether the prediction and the score move closer together. **Kiffer:** Yes. A student whose predictions keep running ahead of their scores has learned something useful about their own judgement, and they can use it to decide where the next study session goes. **Sarah:** Question two. How big was the Harvard tutoring result, and how far should we trust it? **Kiffer:** Let's start with a made-up assistant reply, because it's the kind of thing you might get if you ask about the study. Suppose you ask what an effect size of 0.63 standard deviations means in the trial by Kestin and colleagues, and the reply says that students with the AI tutor scored sixty-three percent higher than students in the class. How does that sit with you? **Sarah:** It doesn't sit well. An effect size of 0.63 means the average student with the tutor scored 0.63 standard deviations above the average student in class. The reply has turned a distance measured in units of the spread of scores into a percentage increase. **Kiffer:** And the medians give you a second check on the size of the gain. Before instruction, the median score on the short test was 2.75. After the class it was 3.5, and after the tutor it was 4.5. **Sarah:** So, using the medians, the gain in class was three quarters of a point, the gain with the tutor was 1.75, and 1.75 divided by 0.75 is about 2.3. And the tutor's median score of 4.5 is about twenty-nine percent above the class's 3.5, so neither comparison comes out at sixty-three percent. **Kiffer:** Right. The 2.3 is where the "more than double" figure comes from. Effect sizes are easier to picture as percentiles, so let's warm up with the famous one. Bloom reported in 1984 that students tutored one to one performed about two standard deviations above students in a conventional class. If scores follow a bell curve, roughly what share of the conventional class would the average tutored student outscore? Think about how much of a bell curve lies within two standard deviations of the average. Take a few seconds. *(Pause)* **Sarah:** About ninety-five percent of a bell curve lies within two standard deviations of the average, so about two and a half percent lies above that point. The average tutored student would outscore about ninety-seven and a half percent of the class. **Kiffer:** That's right. A table gives just under ninety-eight percent. The same conversion takes the average effect of human tutoring in the 2011 review, 0.79 standard deviations, to about seventy-eight and a half percent. And on the same assumption, Kestin's 0.63 puts the average student with the AI tutor above about seventy-four percent of students in the active-learning class. **Sarah:** So on paper the AI tutor did a bit less than the average human tutor. **Kiffer:** That's how it looks on paper. The comparisons differ, though. The 2011 review compared tutoring with no tutoring, and Kestin compared the AI tutor with an active-learning class, which already has strong evidence of benefit over lecturing. The authors also regard 0.63 as an underestimate, because many students reached the top of the test's scale and had no room to show a further gain. Scores piled up at the top aren't shaped like a bell curve either, so the seventy-four percent is only a rough guide. **Sarah:** Okay, here's where I want to push. It's one course at one institution, two introductory topics, students who'd chosen their course, and a tutor built by the lead author. That's a lot of reasons to hold back. **Kiffer:** They're fair, and the authors are cautious themselves. They don't claim the tutor would beat active learning on work that calls for complex synthesis or higher-order critical thinking. What I take from the trial is narrower. A tutor that holds correct solutions, manages how much new information arrives at once, walks students through each part of a problem and gives hints as they work can produce real learning in less time. The median time with the tutor was forty-nine minutes, against about sixty minutes of learning time in class. **Sarah:** And Bastani's tutoring version is the check on that. It also held correct solutions and gave hints, and it only matched practice with a textbook. **Kiffer:** So I'd read the two studies together as saying that a tutoring design is needed for benefit and may not be enough by itself. The Harvard tutor sat inside a carefully built lesson with good problems. I'd treat the trial as a guide to what to put into your own setup, and I wouldn't expect the same effect from a general assistant with no instructions. **Sarah:** Question three. Can you trust a practice quiz when every answer in the key comes with a quotation from your own notes? **Kiffer:** Let's build one. Jordan, the co-op student in the course's fictional Cedar Valley Health Authority case, needs to read the team's reports, which use odds ratios. They've been studying risk ratios and odds ratios, and they paste a short passage from their notes into the practice-question prompt from the lesson, the one that asks for a supporting quotation for each keyed answer. **Sarah:** What does the passage say? **Kiffer:** Three sentences. The risk ratio divides the risk of the outcome in the exposed group by the risk in the unexposed group. The odds ratio divides the odds of the outcome in the exposed group by the odds in the unexposed group. When the outcome is rare, the odds ratio is close to the risk ratio. **Sarah:** And one of the questions? **Kiffer:** It uses made-up numbers. In a cohort, thirty of one hundred exposed people and fifteen of one hundred unexposed people develop the outcome. Option A says the risk ratio is 2.0 and the odds ratio is also about 2.0. Option B says the risk ratio is 2.0 and the odds ratio is about 2.4. Option C says the risk ratio is 0.5. Option D says an odds ratio can't be calculated from a cohort. The key says A, and it quotes the sentence, "When the outcome is rare, the odds ratio is close to the risk ratio." Is the key right? Take a few seconds. *(Pause)* **Sarah:** Let me try this one. First I check that the quotation really is in the passage, and it is, word for word. Then I check the risk ratio. Thirty out of a hundred is 0.3, fifteen out of a hundred is 0.15, and 0.3 divided by 0.15 is two. The quotation says the odds ratio is close to the risk ratio, so the odds ratio is about two as well. I'd accept the key. **Kiffer:** You've done two good checks. Before you accept it, read the quotation once more. What has to be true for it to apply? **Sarah:** The outcome has to be rare. And thirty percent in the exposed group isn't rare at all. So I took the quotation on trust without asking whether it fit the case. Let me work out the odds ratio. In the exposed group, thirty had the outcome and seventy didn't, so the odds are thirty divided by seventy, about 0.43. In the unexposed group it's fifteen divided by eighty-five, about 0.18. And 0.43 divided by 0.18 is about 2.4. **Kiffer:** The cross-product gives the same thing. Thirty times eighty-five is two thousand five hundred and fifty. Seventy times fifteen is one thousand and fifty. Divide the first by the second and you get about 2.43. So the key is wrong, and the answer is B. **Sarah:** What bothers me is that the quotation check passed. The sentence was real, and it was right there in the passage. **Kiffer:** That's what makes this harder than a key with no quotation at all. A quotation can be genuine and still fail to support the key, because it carries a condition the question doesn't meet. So the check has three parts. Is the quotation in the source? Does it apply to this case? And if there's a number, can you get it yourself? **Sarah:** And the harm could last. If Jordan learned from that key, they might read an odds ratio for a common outcome as if it were a risk ratio. It's the same gap you saw in Doll and Hill's 1950 data for men. Smoking was so common that the odds of having smoked were fourteen times as high among patients as among controls, while the ratio of the proportions of smokers was only about 1.04. **Kiffer:** Yes. Whenever the ratio of two proportions is above one, the odds ratio sits further from one, and the gap grows as the thing being counted gets more common. A wrong key teaches the wrong answer with the authority of a marked test, so the two minutes it takes to recompute are well spent. **Sarah:** Question four. Does relying on AI make people weaker thinkers? This one often comes up alongside Gerlich's 2025 survey of six hundred and sixty-six people in the United Kingdom. People who reported using AI tools more often scored lower on a measure of critical thinking, with a correlation of minus 0.68. That's strong. If I square it, I get about 0.46, so AI use and critical thinking scores share almost half of their variation. It's hard to dismiss that. **Kiffer:** I wouldn't dismiss it either. The survey measured everyone once, though, at a single point in time. So here's a question. Apart from AI use wearing down critical thinking, what else could produce that correlation? Take a few seconds. *(Pause)* **Sarah:** It could run the other way. People who are less inclined to think critically might choose to lean on AI more. Or a third factor could drive both, and age is the obvious candidate, since younger participants reported heavier use and more offloading, and they also scored lower on critical thinking. **Kiffer:** Both of those fit the data as well as the causal reading does. Education is worth checking too. Participants with more education scored higher on critical thinking whatever their AI use, so if education also differs between heavy and light users, it could account for part of the correlation. And there's the measurement. Both scores came from one questionnaire, much of it self-report, and when two scores come from the same questionnaire filled in by the same person, part of the correlation can come from how that person answers surveys. That's one reason a correlation this strong deserves a second look. **Sarah:** So what should a student who's worried about this actually do with it? **Kiffer:** I'd treat the survey as a reason to pay attention and a source of hypotheses, and let the randomized studies carry the causal argument. Then I'd use the idea of cognitive offloading. Offloading is ordinary, and writing a shopping list is an example. The question is whether the part you hand over is the part you're trying to learn. If Jordan asks an assistant to fix the spelling and layout of their study notes, nothing is lost. If they ask it to interpret a confidence interval while they're learning to interpret confidence intervals, they've handed over the skill itself. **Sarah:** So the same request can be fine for one person and costly for another. **Kiffer:** Right. An analyst on the team who already reads confidence intervals fluently loses little by having an assistant draft the interpretation, as long as they check it. Jordan, who is still learning, would lose the practice. **Sarah:** Okay. Now the part where we disagree. **Kiffer:** This one's about hints. **Sarah:** Here's my worry. A hint is still help, and every hint takes away a little of the effort that makes practice work. Bastani's tutoring version gave hints and produced no gain on the unassisted test. One possible reason is exactly that, that hints still reduce the effort a student has to make. **Kiffer:** That's a fair reading, and there are other possible reasons too: only four sessions, and a comparison group that was already doing a lot of retrieval with their textbook and notes. Where I'd push back is on the idea that more struggle is always better. Kestin's tutor gave hints as students worked, and they learned more in less time. A student who's been stuck for forty minutes with no idea of the next step can easily give up or go looking for the answer, and then the struggle is gone anyway. **Sarah:** But some of that struggle is what makes the practice work. Desirable difficulties are supposed to feel hard. If a hint arrives every time it gets hard, the useful difficulty is the first thing to go. **Kiffer:** Retrieving something, or working out the next step from a hint, is the useful kind of hard. Sitting with no idea where to start can be a different kind. A good hint names the relevant idea, and the student still has to do the step. That's a long way from the general assistant in the Bastani study, where the most common first message was "what is the answer?" **Sarah:** That works in principle. In practice, a tired student late at night can ask for four hints in a row, and by the fourth one they've been walked through the whole problem. A tutor responds to what you ask for, the same as the built-in learning modes. If you press, you'll usually get the answer. **Kiffer:** That's the strongest version of your argument, and I think you're right about the risk. I'd still keep hints, because they let a student practise on material just past their reach, at any hour, and that's a large part of what makes AI tutoring worth having. What I'd add is a check after the fact. After any problem you finish with hints, solve a matched problem, the same idea with new numbers, with the tutor closed. If you can do it, the hints helped you learn. If you can't, the hints carried you, and you know where to go back. **Sarah:** I'd still set a higher bar for asking than you would. I'd want a written attempt at the step first, something on the page, even if it's wrong. Producing that attempt is the generation effect at work, since people remember what they produce themselves better than what they read. **Kiffer:** Then let's put both into one rule. Write down an attempt at the step before you ask for a hint, ask for one hint at a time, and after any problem you finish with hints, solve a matched one unaided. **Sarah:** I'll probably still ask for fewer hints than Kiffer would, and I'd follow that rule. **Kiffer:** Let's pull it together. Here are three things to take away. **Sarah:** First, practice with an assistant shows what you can do with help. To find out what you've learned, test yourself with the assistant closed, and compare the score you predicted with the score you got. **Kiffer:** Second, check a generated answer key in three steps. Find the quotation in the source, ask whether it applies to the case in the question, and recompute any number yourself. **Sarah:** And third, read the evidence on AI and learning for what each design can show. The randomized studies tell you that who does the thinking matters, and a survey taken at one point in time can't tell you which way an association runs. **Kiffer:** If you'd like more practice, rework the quiz question on risk ratios and odds ratios from this episode with a rare outcome, say six in a thousand exposed people and three in a thousand unexposed, and see how close the odds ratio comes to the risk ratio. **Sarah:** Next time, it's Lesson five, AI-Assisted Literature Searching and Reading, where the reading techniques from this lesson are put to work finding and checking evidence. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson five.