# Lesson 2: How Large Language Models and AI Agents Work *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** When GPT-2 predicts the most common cause of death in Canada, it puts heart ahead of cancer. **Kiffer:** And Statistics Canada's 2024 data put cancer first. So between those two, GPT-2 prefers the wrong one. **Sarah:** Then turn the temperature down and ask again. A lot of students believe a lower temperature makes a model more accurate. **Kiffer:** GPT-2 puts heart ahead of cancer at every temperature. To see why, you have to look closely at what those probabilities describe, and at what temperature does to them. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. This is Lesson two, How Large Language Models and AI Agents Work, and Sarah's got the first question. **Sarah:** Question one. What does a next-token probability actually tell you? I want to begin with a phrase people use about these tools all the time, which is fancy autocomplete. Is that fair? **Kiffer:** As a description of the mechanism, it's fair. The model writes one token at a time, each chosen from a probability for every token in its vocabulary. As a prediction of what the tools can do, it undersells them. To predict the next token well across trillions of tokens of text, a model has to encode grammar, facts and patterns of reasoning in its weights. **Sarah:** I'd defend the phrase a little, though. It reminds people that the model is producing likely text, with no separate step that checks whether the text is true, and that keeps them checking. **Kiffer:** That's its value. The most useful version of the description is the one that predicts something. These tools are strong where the information they need is abundant in their training text or supplied in the prompt, and weak on rare or recent facts they have to recall. **Sarah:** So let's look at the probabilities themselves, with another GPT-2 prompt we haven't talked through. Loneliness among older adults is associated with an increased risk of. At temperature one, GPT-2 puts suicide first at about seven percent, developing second at about six percent, depression third at about six percent, and dementia fourth at about four percent. **Kiffer:** Suicide is a serious subject, so it's worth being careful about what that number means. It's tempting to read the list as GPT-2 ranking the health effects of loneliness. What the seven percent describes is text. It's GPT-2's estimate, learned from web pages, of how often that phrase goes on with the token suicide, about seven times in a hundred. It's a statement about the next piece of a sentence. Whether loneliness raises any of these risks, and by how much, is a question for studies of older adults. **Sarah:** And the second token, developing, isn't a health effect at all. **Kiffer:** Right, it's a word that leads into one. A sentence that goes on to say developing depression, or developing dementia, puts its probability on developing first, and the condition comes a token later. So the probability for one idea can be spread over several tokens and several phrasings. The top bar tells you the most probable next token. It doesn't tell you the model's top idea, and it tells you nothing about the strength of the evidence. **Sarah:** Okay, now a calculation, using the ratio rule from the lesson. At temperature one, depression has a probability of 0.0612 and dementia 0.0393, so depression is about 1.56 times as probable as dementia. If we lower the temperature to 0.5, what does that ratio become? I'll try it myself in a moment. Take a few seconds. *(Pause)* **Sarah:** Here's my attempt. Dividing by 0.5 is the same as doubling, so the ratio doubles, from 1.56 to about 3.1. At temperature 0.5, depression should be about 3.1 times as probable as dementia. **Kiffer:** You can check that against GPT-2's actual values at temperature 0.5. What are they? **Sarah:** Depression is 0.1875 and dementia is 0.0776. That's a ratio of about 2.4. So 3.1 is too high. **Kiffer:** That comparison is worth doing every time you can. So which step went wrong? **Sarah:** The doubling. Dividing by 0.5 doubles the gap between the two scores. The softmax turns a gap between scores into a ratio of probabilities through an exponent, so doubling the gap squares the ratio. That's the rule from the lesson, raise the ratio to the power one divided by the temperature, and one divided by 0.5 is two. 1.56 squared is about 2.4, which matches. **Kiffer:** Exactly. And look at the probabilities themselves. Depression's probability tripled, from about six percent to about nineteen, and dementia's roughly doubled, from about four percent to about eight. Lowering the temperature pulled probability out of the long tail and onto the leading tokens, gave the biggest gains to the tokens that started highest, and kept them all in the same order. **Sarah:** Which matters, because a lot of students believe that turning the temperature down makes a model more accurate. **Kiffer:** It makes it more consistent. Temperature never changes the order of the tokens, so whatever the model ranks first stays first. GPT-2 puts heart ahead of cancer for the most common cause of death in Canada at every temperature, and Statistics Canada's 2024 data put cancer first. A low temperature gives you the model's favourite answer more reliably, whether that answer is right or wrong. **Sarah:** Here's a harder one that combines sampling with a bit of probability. After the prompt, The vaccine was shown to be, GPT-2 gives ineffective a probability of 0.0142 at temperature one. That's about one sample in seventy. Suppose, as a thought experiment, we drew fifty separate samples from that prompt at temperature one. What's the chance that at least one of the fifty continues with ineffective? Take a few seconds. *(Pause)* **Kiffer:** The trap is to multiply. Fifty times 0.0142 is 0.71, and it's tempting to call that a seventy-one percent chance. That number is the average count of ineffective samples in a set of fifty. It can't be a probability, because with two hundred samples it would come to 2.84, which is more than one. **Sarah:** So we go through the chance of none. **Kiffer:** Right. Each sample avoids ineffective with probability one minus 0.0142, which is 0.9858. All fifty avoid it with probability 0.9858 to the power fifty, which is about 0.49. One minus 0.49 gives about 0.51. **Sarah:** So it's about a coin flip. One in seventy sounds small, and over fifty draws it's even odds that the opposite claim shows up at least once. Would a higher temperature make that worse? **Kiffer:** For that one word, it makes it slightly better. At temperature 1.5, ineffective falls a little, to 0.0131. Compared with effective it gains a lot, going from about one twenty-sixth as probable to about one ninth. The leading tokens lose so much, though, that the top ten together drop from about eighty-two percent to about thirty-one. Over fifty samples, the chance of at least one ineffective comes out at about forty-eight percent, against about three and a half percent at temperature 0.5. The cost at 1.5 is that about seven samples in ten continue with one of the tens of thousands of tokens outside the top ten, so the text gets stranger in other ways. **Sarah:** Nobody drafts vaccine messages with GPT-2, though. What do we take from this for a real assistant? **Kiffer:** A modern assistant is far more reliable than GPT-2, and it still writes by the same mechanism. The point is the arithmetic of rare events. An error with a small chance in any one answer becomes close to certain somewhere across thousands of answers, and a team that uses an assistant every day is making thousands of draws. That's why checks have to be routine for every factual claim that goes out, including the answers that look fine. **Sarah:** Question two. Can you instruct a model to stop making things up? I see prompts all the time that end with something like, only cite real, peer-reviewed sources, and don't invent anything. **Kiffer:** Let's try one. This example is made up. A student asks an assistant, with web search off, for three peer-reviewed studies on social prescribing for older adults in Canada, and adds, only cite real sources. The reply lists three references. Each has authors, a year, a journal, a volume and page numbers, and a sentence summarizing the findings. It looks exactly like the reference list of a review article. **Sarah:** And the instruction does nothing? **Kiffer:** It can change the manner. A well-trained assistant may add a note telling you to verify the references, or say it can't be certain they exist. What an instruction can't do is supply knowledge. It's more text in the context window. If the model never absorbed a particular study well enough in pretraining, the instruction gives it nothing to recall, and the most probable continuation of a reference list is something shaped like a reference. **Sarah:** So while it's writing, the model can't reliably tell a real reference from an invented one. **Kiffer:** Right. It keeps no record of where its patterns came from. And Kalai and colleagues argued in 2025 that the way models are trained and evaluated rewards guessing, because a test that only counts right answers gives a guess more credit than an honest I don't know. A polite line at the end of your prompt is pushing against all of that. **Sarah:** Here's my skeptical question, then. If an instruction can't add knowledge, why do organizations put rules like that in their system prompts? The fictional Cedar Valley Health Authority's system prompt tells its assistant to say so when it isn't confident a fact is correct. **Kiffer:** Because it helps at the margin. Assistants are trained to give the system prompt priority, and they usually follow it, so a model asked to flag uncertainty will flag it more often. Its sense of its own uncertainty is weak, though, because the probability of a fluent sentence is a different quantity from the probability that its claim is true. The rule reduces the problem, and the team still checks. **Sarah:** So what actually helps is changing what's in the context. Turn web search on, or upload the studies you already have and ask the assistant to work only from them. That turns a recall task into a reading task. **Kiffer:** And reading is where these tools are strong. Then open every reference before it goes anywhere. The four-step citation check in Lesson five is built for this. **Sarah:** Question three. When an answer cites a real source, what can still go wrong? I find this one the hardest, because a working link feels like the end of the checking. **Kiffer:** It's where the checking starts, because retrieval changes the kind of error you get. Here's a made-up example, using an invented report from the fictional authority. Jordan, the co-op student on the Cedar Valley team, asks the assistant what share of residents got a flu shot last season. The assistant searches and retrieves a passage from the report that reads, in the 2025 to 2026 season, influenza vaccine uptake among Cedar Valley residents aged sixty-five and older was sixty-one percent, up from fifty-eight percent the season before. Uptake among younger adults was not measured. **Sarah:** And what does the answer say? **Kiffer:** It says, flu vaccine uptake in Cedar Valley rose to sixty-one percent last season, and it cites the report. The citation is real and the number is in the passage. What's wrong with the answer? Take a few seconds. *(Pause)* **Sarah:** It dropped the population. Sixty-one percent applies to residents aged sixty-five and older, and the answer makes it sound like everyone. The passage also says uptake in younger adults wasn't measured, so there's no figure for all residents at all. **Kiffer:** That's it. A real source reported inaccurately is the characteristic error of retrieval, the same kind of slip as the dementia sentence in Jordan's search-on answer. Small models like the one inside Lewis are prone to it, which is why every citation in Lewis is a button, and large models do it too. The check is to read the cited sentence and compare its subject, population, place and date with the answer. **Sarah:** There's also the retrieval step itself. Lewis ranks passages by the words they share with the question. So if I type, why does the chatbot make stuff up, I'm using words the lessons mostly don't use. **Kiffer:** Good example. Keyword ranking with BM25 rewards shared words, and the lessons say fabrication or hallucination. That question might get a weak match and the not-covered reply, or it might pull up a passage that happens to mention chatbots. Rephrasing it in the lesson's terms, why do language models fabricate references, gives the ranking something to work with. Embedding retrieval copes better with that kind of rewording, which is one reason many systems combine the two. **Sarah:** And with a long document I upload myself, there's the problem of the middle. **Kiffer:** Yes. Liu and colleagues found in 2024 that models answered more accurately when the relevant information sat near the beginning or end of a long input than when it sat in the middle. So a forty-page report that fits easily in the context window can still be read unevenly. If the table you care about is on page twenty, I'd paste that section in on its own, or ask about that table directly and check the answer against the page. **Sarah:** Question four. When a task goes to an agent, where should a person step in? I want to try a design that looks safer than the one with the hidden instruction on a web page. **Kiffer:** Go ahead. **Sarah:** This one's made up. The Cedar Valley team gets a lot of email from the public in a general inbox. They're considering an agent that reads each incoming message, looks up the right staff member in the team's internal directory, and writes a draft reply. It can't send anything. The drafts sit in a folder until a staff member sends them. Of the three dangerous ingredients, private data, untrusted content and a way to send information out, which ones does this design have? Take a few seconds. *(Pause)* **Kiffer:** I count two and a half. Untrusted content, yes, because anyone can write to that inbox, so anyone can put instructions in front of the agent. Private data, yes, the internal directory. A way to send things out looks absent, because the agent only writes drafts. **Sarah:** And the half is the drafts. A draft goes out the moment a busy staff member clicks send without reading it closely. Suppose a message hides a line telling the agent to paste the whole directory into its reply. If the reviewer skims, the reviewer becomes the send button. **Kiffer:** Right. In this design the human checkpoint is doing most of the protective work, so it has to be a real check, with time set aside for it. **Sarah:** There's a problem before we even get to injection. People write to a health authority about their health. Some of those emails will contain personal health information, and the pilot's rule is that the assistant never receives any. **Kiffer:** That's the bigger issue, and it settles the question before the security analysis does. A safer version keeps the agent away from the inbox entirely. A staff member reads each message, and when a general question comes up, they paste it in with every identifying detail removed and ask for a draft answer based on the team's public pages. **Sarah:** So the task goes back to a person, with the assistant in a supporting role. **Kiffer:** That's a sensible place for a review like this to end up. Start with the simplest design that does the job, give it only the access it needs, and put a person before anything that leaves the team. **Sarah:** Okay. Now the part where we disagree. **Kiffer:** Public chatbots. **Sarah:** Suppose the Cedar Valley team wanted a chatbot on its website, answering questions like where to get a flu shot or what a program offers. I'm wary. Everything we've said about agents puts a person between the model and anything consequential. A public chatbot puts its answer straight in front of a member of the public, with no review at all. **Kiffer:** That's the strongest objection, and I'd still support a carefully limited pilot. The design would look a lot like Lewis. It asks people up front to leave out personal health details, answers only from pages the team has already approved, shows the source page with every answer, and gives a fixed reply pointing to a phone line or a staff member for anything outside that scope. The review moves earlier. The team approves the pages, tests the bot on a long list of realistic questions before launch, and reviews a sample of conversations regularly. **Sarah:** Reviewing conversations afterward means finding an error after it has already reached someone. And Lewis, which is carefully built, still comes with a warning that its model can misstate a passage. **Kiffer:** True. The comparison I'd make is with what people do now. After hours, someone with a question searches the web and lands on whatever page comes up, some of it outdated. A bot that answers from the authority's own current pages, and shows them, could leave many people better informed, even with occasional errors. **Sarah:** I'd worry that the people least confident reading health information are the ones most likely to trust the bot's summary and never open the page. And there's accountability. In February 2024, British Columbia's Civil Resolution Tribunal decided a case against Air Canada. The airline's website chatbot had told a customer he could apply for a reduced bereavement fare after he'd already travelled, within ninety days of the ticket being issued, and the airline's own bereavement page said the policy didn't cover requests made after travel. Air Canada argued it couldn't be held liable for information given by its agents, including the chatbot. **Kiffer:** The tribunal called that, in effect, treating the chatbot as a separate legal entity responsible for its own actions, and it held that the airline was responsible for all the information on its website, whether it came from a static page or a chatbot. I think that's the right principle, and it matches the idea of a named human owner for every automated use. If a health authority runs a chatbot, the authority owns every answer it gives. **Sarah:** So where do you land? **Kiffer:** I'd pilot it for a narrow set of questions with clear answers on approved pages, such as clinic locations and program details, and keep it away from anything that calls for clinical judgement. Anything that sounds like a crisis gets a fixed reply with nine eight eight and nine one one, the way Lewis handles it. **Sarah:** I'm more skeptical than Kiffer. I'd want the pilot judged partly on how often people actually open the source page, and I'd want a simple way to switch it off. We agree on the practical rule, though. A public chatbot answers only from pages the team has approved, shows the page with every answer, and hands everything else to a fixed reply or a person. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, a next-token probability describes text. Temperature changes how concentrated the choices are and never changes their order, so a low temperature makes a model more consistent without making it more accurate. **Kiffer:** Second, rare errors add up. A one-in-seventy chance per draw is close to even odds over fifty draws, so checking has to be routine for every factual claim that leaves your hands. **Sarah:** And third, each remedy brings its own error. Instructions shape the manner and add no knowledge, retrieval brings real sources that can be misreported, and agents can be redirected by what they read. In every case the check comes back to opening the source and putting a person before anything consequential. **Kiffer:** If you'd like more practice, rework the fifty-sample problem from this episode with the loneliness prompt. At temperature one, GPT-2 gives dementia a probability of 0.0393. Work out the chance that at least one of twenty samples continues with dementia, and compare it with the answer you'd get by multiplying. **Sarah:** Next time, it's Lesson three, Working with AI: Prompting, Context and Verification, which maps the uneven frontier of what these tools do well and sets out a verification routine scaled to the stakes of a task. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson three.