# Lesson 8: Content Analysis *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a made-up finding. In a study of twelve older adults, the men talked about their pets in forty coded passages and the women in twelve. So the men talk about pets more than the women do, right? **Kiffer:** Maybe. Or maybe one man spent most of his interview talking about his dog. Before that forty means anything, somebody has to tell you whether you're counting passages or people. **Sarah:** So the same data could point the other way? **Kiffer:** It certainly can. Choosing what to count is one of the first decisions in a content analysis, and it's also one of the easiest to leave out of a methods section. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. That question, what a number in a content analysis is actually counting, runs through all of Lesson eight, Content Analysis, so let's start with it. **Sarah:** Question one. When a content analysis reports a number, what exactly was counted? Kiffer, give us the rest of the made-up pet study. **Kiffer:** There are six men and six women. Two of the men mentioned pets at all, and one of them, a retired farmer who talked about his dog at length, accounts for twenty-eight of the men's forty coded passages. Four of the six women mentioned pets, with three passages each, which gives their twelve. So here's the question for everyone. Which group talks about pets more, the men or the women? Think about what the unit is in each version of the count. Take a few seconds. Pause the audio if you'd like more time. *(Pause)* **Sarah:** It depends on the recording unit, and the two versions point in opposite directions. Counted by passage, the men lead forty to twelve. Counted by person, four of six women raised pets compared with two of six men. That's sixty-seven percent compared with thirty-three percent, so the women lead. And most of the men's total comes from one talkative participant. **Kiffer:** Right, and both are correct arithmetic for different questions. The count by person asks how many participants raised the topic at all. The count by passage mixes that together with how long people talk, how hard the interviewer probed and how finely the coder divided the text. If you want to compare groups of people, the count by person is usually the one that matches the claim. **Sarah:** And what if somebody ran a chi-squared test on the passage counts? **Kiffer:** Then there's a second problem. Twenty-eight passages from one man share a speaker, a mood and an interviewer, so they carry much less information than twenty-eight separate people would. A test that treats them as independent usually gives a p-value that's too small. In an interview study the participant is usually the independent unit, which is one reason transcript-level recording units suit a small corpus. **Sarah:** There's a related misreading I see all the time. A code turns up in eighteen of twenty transcripts, and somebody writes that ninety percent of lonely people describe loneliness in bodily terms. **Kiffer:** Eighteen of twenty describes this corpus: these participants, recruited this way, answering these questions. With a purposive sample, the percentage tells you how widely an idea ran through the people you chose to talk with, and its meaning stops at the edge of that group. Things change when the texts themselves are what you want to describe. **Sarah:** A Canadian study of opioid crisis coverage in The Globe and Mail fits that description. A team published it in 2020, covering the paper's print articles from 2000 to mid-2018, six hundred fifty articles in all. Some of what they tracked sits near the manifest end, like stigmatizing terms such as junkies and abuser, which declined from 2009 on. Some was latent, like who an article held responsible. Only twelve articles, about 1.8 percent, mentioned root causes such as social inequity or unresolved trauma. **Kiffer:** And there the percentage means something directly, because the question is about what that newspaper published. The authors were also clear about the limit: it's one newspaper's print articles, and only part of the population reads a printed paper. The count matches the claim because the claim stays inside the corpus. **Sarah:** Question two. How do you write a latent code that a second coder can apply without coming back to ask what you meant? I drafted one, so I'll be the test case. It's invented for this episode. The label is role-loss loneliness. The definition is loneliness the participant links to the end of a role, such as a job, parenting or caregiving. **Kiffer:** And what did you write for the inclusion and exclusion rules? **Sarah:** Inclusion: code any passage that mentions retiring, a job ending or the kids moving out, together with the word lonely or alone. Exclusion: loneliness tied to the death of a partner, which goes under a separate bereavement code. **Kiffer:** Let's test it on two borderline excerpts, both made up for this episode. Excerpt one: "When I retired from the post office, I worried I'd be alone all day. Turns out I'm busier than ever. I volunteer at the food bank three mornings a week, and I've made real friends there." Excerpt two: "For thirty years I ran the counter at the hardware store. People came in just to talk. These days, some afternoons I stand at the window and wonder who even knows my name anymore." **Sarah:** Okay, applying my rule. Excerpt one mentions retiring and uses the word alone, so it's coded. Excerpt two never mentions retiring, a job ending, lonely or alone, so it stays uncoded. **Kiffer:** Here's a question for everyone listening. Did Sarah's rule get those two excerpts right? Take a few seconds. *(Pause)* **Sarah:** It got them backwards. The first speaker is telling us the worry never came true. She's describing a retirement full of new friends. The second speaker is the clearest case of role-loss loneliness I could ask for, and my rule threw him out because he never used the words. **Kiffer:** So where did the definition itself go wrong? **Sarah:** The inclusion rule is a word list. I gave the code a latent label and then wrote a manifest test for it. Anyone following my rule would be matching words. **Kiffer:** That's a very common slip with latent codes, and the tension behind it goes back a long way. Berelson's 1952 definition kept content analysis to manifest content and paired that with objectivity, meaning other analysts could repeat the work. Later work, most influentially Krippendorff's, opened the method to latent meaning, with the expectation that you show latent codes can be applied consistently. So how would you rewrite the inclusion rule? **Sarah:** Code a passage when the participant connects a present sense of disconnection, of being unneeded or unknown, to the end of a role that used to bring them into contact with people, whether or not words like retire, lonely or alone appear. And the exclusion rule gets a second part. A role ending described with no loss of connection, like a retirement that brought new friends, isn't coded. **Kiffer:** Good. Now give it examples. The hardware store passage is your typical example. For an atypical one, picture a caregiver whose mother has just moved into long-term care, saying the house is quiet and nobody needs her at seven in the morning anymore. Nobody retired and her mother is still alive. A role has still ended, and the definition fits. **Sarah:** And the post office excerpt goes into the codebook as a near miss, so the next coder can see something that looks similar and doesn't count. **Kiffer:** One more test. Suppose a coder saw only the last sentence of the hardware store excerpt, the one about standing at the window. Could they apply your code? **Sarah:** No. From that sentence alone you'd know he feels unknown, and you wouldn't know a role had ended. The thirty years at the counter is two sentences earlier. So the context unit has to be at least the paragraph. **Kiffer:** Question three. What does an agreement check tell you, and what can't it tell you? Let's stay with Sarah's revised code and an invented check. Two coders work through the same transcript in a program where each coder highlights their own passages and tags them. Coder one highlights six passages with the role-loss code, and coder two highlights five. Four of coder one's passages line up with four of coder two's. **Sarah:** So if we line up those four matching pairs, both coders chose role-loss loneliness every time. That's a hundred percent agreement. **Kiffer:** Here's the question for everyone. What's missing from that hundred percent? Take a few seconds. *(Pause)* **Kiffer:** What's missing is everything one coder marked and the other didn't. Coder one has two passages with no partner, and coder two has one. Those are the biggest disagreements in the set, about whether the code is there at all, and pairing up matched highlights quietly drops them. Count every passage either coder marked and you have seven, with agreement on four of them, which is about fifty-seven percent. **Sarah:** And even the matched pairs might not match very well. If coder one highlighted one sentence and coder two highlighted a whole two-minute answer, they've agreed on the code and disagreed on the unit. **Kiffer:** Right. That's a unitizing disagreement, about where a coded stretch of text starts and stops, and it matters as soon as you count passages or words. Krippendorff developed agreement coefficients for unitizing itself, and there's also a simpler fix that goes back to question one. Fix the recording units before anyone codes, say paragraphs or speaker turns, so both coders make a yes or no decision on every unit and every disagreement gets counted. **Sarah:** And at the transcript level the boundaries stop mattering. Both coders only have to agree on whether the code appears anywhere in the transcript. **Kiffer:** That's the trade behind transcript-level units in a small corpus. You get clean decisions, and you give up any information about how often a code comes up within an interview. **Sarah:** That brings up something that bothers me about small interview studies. Say the recording unit is the whole transcript, and you double-code ten percent of twenty transcripts. **Kiffer:** Then you have two transcripts, and each code gets two yes-or-no decisions. Two decisions can't give you a meaningful estimate of agreement. With a corpus of twenty, I'd have a second coder do much more than ten percent, and often all of it, so the alpha has enough decisions behind it to mean something. **Sarah:** And here's the part that stings. My first keyword rule would probably have produced a beautiful alpha. **Kiffer:** Very likely, because two people matching words will agree almost every time. Agreement shows that a code is applied consistently. Whether the code captures the meaning it names is a separate question, and you answer it by reading what got coded, near misses included. **Sarah:** That worries me more generally. If a high alpha is the visible reward, doesn't the whole field drift toward codes that are easy to agree on and shallow? **Kiffer:** It's a real pressure. The middle band of the thresholds helps here. A latent code that lands between 0.67 and 0.80 can still be reported, flagged as tentative, with the disagreements discussed. And if a code still can't be applied consistently after a couple of rounds of revision, that can be worth reporting as a sign the idea is contested in the data. What I'd avoid is quietly swapping the code for a word list and keeping the old label. **Sarah:** Question four. When does a trend in a content analysis mean what it seems to mean? I'll set this one up, and it's an invented study. A team codes newspaper articles about loneliness for whether they frame it as a public-health problem. In the two years before the pandemic, forty-four of one hundred articles use that frame. In the first two years of the pandemic, seventy-two of two hundred do. So the share falls from forty-four percent to thirty-six percent, just when you'd expect it to rise. **Kiffer:** Now split the corpus by outlet. Before the pandemic, a national paper ran eighty of the articles and forty used the frame, which is half. Local papers ran twenty, and four used it, which is a fifth. During the pandemic, the national paper ran forty articles and twenty-four used the frame, which is sixty percent. The local papers ran one hundred sixty, and forty-eight used it, which is thirty percent. **Sarah:** So here's the question for everyone. The frame became more common in the national paper and more common in the local papers. How can it become less common overall? Take a few seconds. *(Pause)* **Sarah:** The mix of articles changed. Before the pandemic, eighty percent of the articles came from the national paper, which uses the frame more. During the pandemic, eighty percent came from the local papers, which use it less. So the overall share mostly tracks who was doing the writing. **Kiffer:** That's it. Statisticians call this Simpson's paradox, where a pattern that holds inside every group reverses when the groups are combined. It's an easy trap in content analysis, because a corpus is often defined as everything that matches a search, so whichever outlet publishes the most gets the most weight. **Sarah:** So what should the team report? **Kiffer:** The comparison within each outlet comes first. If they want a single overall figure, they can hold the outlet mix fixed. Picture the pandemic years with the old mix, eighty national articles and twenty local ones in every hundred. At the pandemic rates, sixty percent of eighty is forty-eight, and thirty percent of twenty is six, so that's fifty-four in a hundred, or fifty-four percent. Compared with forty-four percent before the pandemic, that's a rise of ten percentage points. **Sarah:** That's the same logic as age standardization, with outlets playing the part of age groups. **Kiffer:** It's the same logic, and the deeper fix is in the sampling design. If the question is whether framing changed over time, sample the same number of articles from each outlet in each period, or analyse the outlets separately from the start. The Globe and Mail study has a quiet advantage here. With one paper, a shift in the outlet mix can't produce a trend, though a shift in article types, say from news to opinion pieces, still could. **Sarah:** Let me bring us back to interview studies, because there's something Kiffer and I see differently. I'm skeptical that a study of twenty or so purposively sampled interviews should report p-values at all. **Kiffer:** Go ahead and make the case, because I think it's a strong one. **Sarah:** A p-value sounds like a statement about a population. Readers see the word significant and hear that it's true of caregivers everywhere. Then there's multiplicity. Eleven codes crossed with, say, four participant variables give forty-four possible comparisons, and at a cut-off of 0.05, chance alone can easily hand you one or two significant results when nothing is going on. In 2016 the American Statistical Association warned that conclusions shouldn't rest only on whether a p-value crosses a threshold. I'd report the counts with their denominators and let the interpretive reading carry the weight. **Kiffer:** I agree with every one of those cautions. Where I differ is on dropping the test altogether. Without one, people read tables by eye, and the eye is generous. Take the hypothetical shame split from the lesson, nine of ten caregivers compared with four of ten non-caregivers. It looks decisive, and Fisher's exact gives about 0.06. The test is what tells you to call that suggestive. Used on a within-corpus question, it works as a brake on overclaiming. **Sarah:** Only if people use it as a brake. A test answers whatever table you hand it. Give it the pooled newspaper table and it will answer a question about a mixture of outlets, which nobody meant to ask. **Kiffer:** That's fair, and no test can tell you that you've built the wrong table. So my position is narrower than it might sound. I'd keep the test for the one or two comparisons you named in writing before the analysis, and I'd call everything else exploratory. **Sarah:** I can work with that, as long as the reading stays at the centre. So here's the rule we share. Report every code with its count and its denominator. Name the one or two comparisons in writing before you run the analysis. Test only those, leading with Fisher's exact and the two proportions. Label everything else exploratory, say plainly that the result describes this corpus, and keep the interpretive memo right next to the numbers. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, every number in a content analysis rests on a unit. Say whether you counted people, passages or words, and run any test on a unit that's independent, which in an interview study is usually the participant. **Kiffer:** Second, a latent code needs a rule written around meaning. Test it on near misses and on passages that use none of the obvious words. High agreement shows consistency, and reading the coded passages is how you find out whether the code captures what it names. **Sarah:** And third, before you trust a trend or a difference in a pooled corpus, split it by whatever changed in its make-up, such as outlet or year, and check that the pattern holds inside each part. **Kiffer:** If you'd like more practice, go back to the role-loss loneliness code from this episode. Write your own inclusion and exclusion rules, then invent two borderline passages to test them, one near miss and one clear case that uses none of the obvious words. **Sarah:** Next time, it's Lesson nine, Schema and Narrative Analysis, where the focus moves from counting codes across many transcripts to tracing the structures inside individual accounts. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson nine.