# Lesson 12: Computational Text Analysis, Cultural Domain Analysis and LLM-Assisted Coding *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's an invented scenario. You paste twenty interview transcripts about loneliness into a language model and ask for a summary. It comes back in seconds, neatly organized, with a quotation from an older participant: "The silence is the loudest thing in my apartment." You search all twenty transcripts, and nobody said it. **Kiffer:** And the unsettling part is that the fabricated quotation reads exactly like the real ones. It fits the themes, it sounds like the people in the study, and nothing in the summary marks it as different from the lines that are real. **Sarah:** So the most quotable line in the summary is the one with nothing underneath it. **Kiffer:** Which is why every result a computer hands you, from a word count to a summary, needs the same first question. What is this actually built on, and can I trace it back to the transcripts? **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. That question of what a computer-generated result is actually built on runs through all of Lesson twelve, Computational Text Analysis, Cultural Domain Analysis and LLM-Assisted Coding, the last lesson of the course, so let's start with one of the simplest results there is, a keyword list. **Sarah:** Question one. When does a keyness result become a finding? Here's an invented example. A keyness comparison finds the word tired forty-eight times in the four older participants' transcripts and eight times in the four younger participants' transcripts, and the test says the difference is highly significant. The older participants' transcripts add up to sixteen thousand words, and the younger participants' to eight thousand. How often does each group say tired per thousand words? Take a few seconds. Pause the audio if you'd like more time. *(Pause)* **Kiffer:** Forty-eight divided by sixteen is three per thousand words for the older group, and eight divided by eight is one per thousand for the younger group. The raw counts make tired look six times as common in the older group, and the rate is three times as high. The test itself allows for the totals, but people reading a table of raw counts often don't. **Sarah:** Three times is still a big difference. **Kiffer:** It is, so let's check where the uses come from. Suppose one older participant said tired forty times in her own transcript, which runs to six thousand words. Can you work out what the other three older participants look like on their own? **Sarah:** Okay. Take her out, and the older group has eight uses left, in sixteen thousand minus six thousand words, which is ten thousand. Eight divided by ten is 0.8 per thousand. That's a little lower than the younger group's one per thousand. So without her, the difference disappears. **Kiffer:** Right. What looked like a group keyword was her word. Keyness tests treat every word in a subcorpus as an independent draw, but in real interviews, words cluster inside particular speakers, so a word concentrated in one transcript can look highly significant for a whole group. With four transcripts per group, that's a serious risk. For every keyword, I'd check how many transcripts it appears in and how much of the count each one contributes. **Sarah:** So is the result worthless? **Kiffer:** The group claim is gone, but her transcript is worth reading. In our invented case, a keyword-in-context list of her forty uses might show that most of them are about caring for her husband through the night. Once you've read those passages in full, that's a promising case-level finding, and a TF-IDF ranking of her transcript, which looks for what's distinctive about one document, is the better tool for describing it. It belongs in the write-up as part of her account, and it tells you very little about older participants in general. **Sarah:** What about the semantic network? There the clusters seem to come straight from the participants' language. Nobody imposes them. **Kiffer:** The decisions are harder to see, but they're there. Somebody chose the window, the stopword list, the fifty words that made it into the network, and the algorithm. Widen the window from a sentence to a whole transcript and the communities blur into one. A talkative participant can also pull a community toward her own vocabulary. A Louvain community is a candidate theme shaped by all of that, and someone still has to read the passages where the words actually occur together. **Sarah:** Question two. What does Smith's salience actually measure? I'd like to try this one myself, because I think it's easy to get wrong. **Kiffer:** Here are five invented free lists, answers to "What helps when you're lonely?" Person one says friends, music, the dog. Person two, walking, friends. Person three, family, church, friends, TV. Person four, a phone-pal program, then family. And person five, family, friends, radio, walking, cooking. **Sarah:** Okay. On each list, an item scores the list length minus its rank plus one, all divided by the list length. Friends is first of three on list one, so it scores one. Second of two on list two, that's one divided by two, 0.5. Third of four on list three, two divided by four, 0.5. And second of five on list five, four divided by five, 0.8. That adds up to 2.8. **Kiffer:** Those four scores are right. Now finish the calculation for friends, and then do the phone-pal program. **Sarah:** Four people listed friends, so 2.8 divided by four gives 0.7. Now the phone-pal program. One person listed it, first on a list of two, so it scores one, and one divided by one is one. So the phone-pal program has a salience of one, the maximum, higher than friends. Hmm. That can't be right. One person out of five mentioned it. **Kiffer:** Your sanity check is doing its job. A salience of one is supposed to mean that everyone named the item first. So here's the question for everyone. What is the correct salience of the phone-pal program, and why? Take a few seconds. *(Pause)* **Sarah:** The formula divides by the total number of participants, all five, and anyone who didn't list an item contributes zero. So the phone-pal program is one divided by five, which is 0.2. And friends is 2.8 divided by five, which is 0.56. **Kiffer:** Right. Dividing only by the people who listed an item throws away half of what salience measures, which is how widely an item is shared. Family, for comparison, is named by three people, first on two of their lists, and comes to 0.5. So friends edges out family, and the phone-pal program falls from the top of the ranking to fourth, behind walking as well. **Sarah:** Does that mean the phone-pal program doesn't matter? **Kiffer:** It means it isn't part of what this group thinks of first. It might matter a great deal to person four. Salience measures how widely shared and how readily recalled an item is, and it says nothing about whether the item works. For someone designing an intervention, a low-salience item that works is exactly what a campaign would need to introduce. **Sarah:** That also explains the "What helps?" lists from the loneliness interviews, where therapy is named by almost as many participants as peer groups, nine compared with ten, but sits further down the lists, so its salience is much lower. There's another catch I'd raise, though. Those lists were extracted from answers in semi-structured interviews. There, the order people mention things depends on what came up earlier in the conversation. **Kiffer:** That's why extraction from interviews is only a second-best approach. If a participant spent ten minutes talking about her daughter before the "What helps?" question, she might skip family as already covered, or name it first because it's on her mind. Rank is the part of salience that interviews distort most. If salience is central to your question, I'd run a proper free-listing task with a single prompt and record the order as it comes. **Sarah:** Question three. If a group agrees, does that make them right? This is about consensus analysis, and I think the word competence misleads people. **Kiffer:** It does. In the Romney, Weller and Batchelder model, a person's competence is estimated from how closely their answers match everyone else's. The answer key the model estimates is the culturally correct answer, the one the group shares. Whether that answer matches the evidence is a separate question. **Sarah:** Here's a Canadian case where that gap could show. Imagine a group whose shared model says loneliness is mainly a problem of old age. They could agree strongly with one another. But Statistics Canada's Canadian Social Survey in 2021 found that twenty-three percent of people aged fifteen to twenty-four said they were always or often lonely, the highest of any age group, compared with nine percent of people aged sixty-five to seventy-four and fourteen percent of those seventy-five and older. **Kiffer:** That survey was collected in August and September 2021, during the pandemic, so I wouldn't lean too hard on the exact figures. The pattern still makes the point. High consensus and high competence scores tell you a model is shared, and a shared model can be out of step with the data. That's useful to a health communicator, because it shows which belief a campaign has to address. **Sarah:** Let's do the eigenvalue ratio next, because it's the number people find hardest to read. Give us some invented numbers. **Kiffer:** Twenty-four participants answer twelve true-or-false statements about loneliness. The first eigenvalue of the agreement matrix is 5.1 and the second is 2.2. What do you conclude, and what would you do next? Take a few seconds. *(Pause)* **Sarah:** 5.1 divided by 2.2 is about 2.3. That's above two and below three, so the support for a single shared model is marginal. I'd treat the estimated cultural answers with caution and look for subgroups, and with loneliness, age is an obvious place to start. **Kiffer:** Suppose you split by age. The older participants give a ratio of about 4.6, and the younger ones a ratio of about four. Each group shares a model, and the two models differ. **Sarah:** So the pooled analysis looked weak because it was averaging over two cultures. And I'd guess the statements they split on are the interesting part. In our invented data, maybe it's something like "talking with people online counts as real contact," where the younger group says true and the older group says false. **Kiffer:** That would make the split itself the finding. It would also line up with the age contrast in the lesson's keyness list, words like screen and online for the younger participants and walker and visit for the older ones, and since it would come from a completely different method, that agreement would be more convincing. A low ratio tells you something about how the domain is organized in your sample. **Sarah:** Question four. Prompt drift sounds like a housekeeping problem. How much damage can one changed phrase in a codebook prompt really do? **Kiffer:** Here's an invented case. A team calibrates its prompt, gets acceptable agreement, and codes the older participants' transcripts first. In version one, structural-cause means attributing loneliness to a structural cause, and the model applies it to twelve of one hundred fifty passages. Before the younger participants' transcripts, someone rewords the definition to any mention of a structural factor, such as housing, work or the pandemic. The model applies it to thirty-nine of one hundred fifty. **Sarah:** That's eight percent compared with twenty-six percent. So the draft report says young adults attribute their loneliness to structural causes more than three times as often as older adults. **Kiffer:** Here's the question for everyone. Before anyone re-runs anything, what's wrong with that comparison? Take a few seconds. *(Pause)* **Sarah:** The prompt changed at the same moment the group changed. Every older passage was coded with version one and every younger passage with version two, so there's no way to separate the age difference from the wording difference. **Kiffer:** That's the trap. So the team runs version two on the older transcripts as well, and the model now applies structural-cause to thirty-three of the same one hundred fifty passages. **Sarah:** Thirty-three out of one hundred fifty is twenty-two percent. So the rewording alone took the older group from eight percent to twenty-two, nearly three times as high. Under one prompt, the age gap is twenty-two compared with twenty-six percent, and with one hundred fifty passages per group, a four-point gap could easily be chance. **Kiffer:** And there's a second trap inside the first. The new wording changed what the code means. "My landlord raised the rent" is a mention of housing, and it's an attribution of loneliness only if the participant links the two. Version two is a different code, so the agreement statistic from calibration belonged to version one. Any change to a definition means re-validating it on the calibration set and re-running every transcript under the locked version. **Sarah:** How sensitive are these models to wording, in general? **Kiffer:** More than most people assume. A 2023 study found that changing only the formatting of a prompt, with its meaning left the same, changed one open model's accuracy on some tasks by as much as seventy-six percentage points. So before locking a prompt, I'd run two harmless paraphrases of each definition on the calibration set. A code whose frequency jumps when you reword it without changing its meaning needs a sharper definition before a model applies it. **Sarah:** And the hallucination audit, the check that would catch a quotation like the one in that summary? If thirty out of thirty sampled passages turn out to be verbatim, is the model clean? **Kiffer:** It's encouraging, with two limits. First, thirty checks can't rule out a failure rate of up to about one in ten. That's the rule of three: when nothing goes wrong in a sample of n, the upper end of a ninety-five percent confidence interval for the true rate is roughly three divided by n. Second, a verbatim passage can still carry the wrong code. The audit asks whether the passage exists and whether the code fits it, and only a reader can answer the second part. **Sarah:** Okay, here's where I'd push back on the whole enterprise. For a study the size of the loneliness interviews, twenty transcripts, I doubt a language model belongs in the coding at all. **Kiffer:** Go ahead and make the case, because I'd answer it a little differently. **Sarah:** I have three reasons. First, by the time you've hand-coded five calibration transcripts, revised the prompt, checked each code and audited the output, you've spent most of the time the model was supposed to save. Second, the codes that carry the analysis, like loneliness-as-cost-of-love or existential-meaning, depend on reading a whole life story, its tone and what someone leaves unsaid, and the model is poor at absence. Third, there's consent. In Canada, university research with people follows TCPS 2, the joint ethics policy of the three federal research agencies, and many consent forms were written before anyone pictured transcripts going to a company's servers. **Kiffer:** I agree with most of that for twenty transcripts. Where I'd differ is on scale and on the kind of code. With two hundred interviews, or thousands of open-text survey answers, the realistic choice is often between a validated model and leaving most of the material uncoded. For codes you can check by looking at the words in the passage, I think a disclosed, validated and audited model is a defensible tool. It can also act as a second reader, because the passages it flags that the human missed are worth a look. **Sarah:** But that second reader makes the same mistake every time it meets the same kind of passage, and its behaviour can shift across the whole corpus when someone edits one phrase in the prompt. And the people whose words it misreads are often the ones least represented in its training text, like participants who describe their loneliness partly in another language. **Kiffer:** That's the strongest point, and it's why I'd build the check for bias into the validation itself. Draw the calibration set from your own participants, including the ones whose experience is least like the model's training text, and check agreement within those groups as well. If the model codes one group's transcripts poorly, those transcripts stay with the human coder. And whatever you decide, disclose it. In 2023, the journal Physica Scripta retracted a paper after a research-integrity investigator spotted the words "Regenerate response," a ChatGPT button label, pasted into the text. The publisher retracted it because the authors hadn't declared using the tool. **Sarah:** So we've each moved a little. I'd still default to hand coding for a small interpretive study. **Kiffer:** And I'd still use a model for large corpora and well-defined codes. So here's a rule we can both live with. Decide code by code. Hand a code to the model only when its definition can be checked by looking at the passage, when it passes its own agreement check on enough instances, and when its frequency holds steady under harmless rewording, in every group of participants you care about. Keep theme development, interpretation and the search for what's missing with the human analyst, settle the data governance before anything is uploaded, and report all of it. **Sarah:** I can work with that. It tells a reader exactly which parts of the analysis the model touched, and on a small study it gives you a clear reason to write down if you decide to code everything by hand. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, before you interpret any count, find out who produced the words. Divide by how much each group said, and check how many transcripts a keyword actually comes from, because one talkative participant can carry a whole group. **Kiffer:** Second, read each statistic for what it measures. Smith's salience divides by everyone in the sample and tells you how shared and how readily recalled an item is, and a consensus score measures how widely a belief is shared, which tells you nothing yet about whether it's accurate. **Sarah:** And third, treat the prompt as part of your instrument. Lock it, keep every version, and re-validate after any change. Every tool in this lesson, from the concordance to the language model, points you toward passages, and reading them, deciding what they mean and noticing what's missing stay with you. **Kiffer:** If you'd like more practice, go back to the prompt-version problem from this episode. Suppose re-running version two on the older transcripts had given fifteen of one hundred fifty passages, with everything else the same. Work out the rates, decide what the age comparison now shows, and list what you'd still have to do before reporting it. **Sarah:** This is the last episode of Office Hours for the course, since Lesson twelve is the final lesson. Thanks for working through all twelve lessons with us. **Kiffer:** Take care, everyone. **Sarah:** Thanks for listening, and all the best with the studies you design next.