# Lesson 5: Finding Themes and Building Codebooks *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a made-up study. Twenty people are interviewed about loneliness, and eighteen of them talk about their phones. The write-up says that phones emerged as a major theme. Are you convinced? **Kiffer:** Not yet. The first thing I'd want to read is the interview guide. If question nine asked, how do you use your phone to keep in touch with people, then eighteen out of twenty mostly tells us that people answered question nine. **Sarah:** So the pattern might have come from the interviewer as much as from the participants. **Kiffer:** It might. And the word emerged hides that possibility, because it makes the theme sound as if it rose out of the transcripts on its own. Somebody wrote the questions, somebody coded the answers, and somebody decided to call it a theme. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. The difference between a pattern in what participants said and a pattern the study built runs right through Lesson five, Finding Themes and Building Codebooks, so let's start there. **Sarah:** Question one. If something comes up in nearly every transcript, does that make it a theme? **Kiffer:** Recurrence is where theme-finding usually starts. Repetitions are the first of the twelve techniques, and nearly every analysis begins there. But a count of transcripts can't tell you why something recurs. I can think of three reasons. Participants may share an experience. The interviewer may have asked everyone about it. Or the code may be so broad that it catches almost anything. Only the first one is a pattern in participants' accounts. **Sarah:** I'd add a fourth way a count can mislead. A word can turn up dozens of times and still come from one person. **Kiffer:** Yes, which is why it helps to count transcripts and mentions separately, and to pull a keyword-in-context list before trusting either number. If every occurrence of a word sits in one interview, you've found a striking feature of one person's account. It may matter a great deal, and it still needs evidence from other participants before it can be called a shared theme. **Sarah:** So in our phone study, how would you tell a shared experience from answers to question nine? **Kiffer:** I'd go back to each transcript and mark where phone talk first appears, before or after the interviewer's phone question. Let's invent the result. Say only five of the eighteen raised phones before they were asked. Then I'd read those five closely and compare them with the thirteen who answered the question. **Sarah:** And suppose the five who raised it on their own all describe scrolling at night and watching other people's lives carry on without them, while the thirteen mostly describe the practical side, who they text and when. **Kiffer:** Then the interesting pattern is in the five. Something like, when people raised it unprompted, the phone was described as a window onto lives going on elsewhere. The thirteen answers are still data, but they're answers to a prompt, and the write-up should say so. **Sarah:** Five out of twenty sounds thin for a theme, though. **Kiffer:** It can still be one. Braun and Clarke made this point in their 2006 paper on thematic analysis. They argued that what makes a theme key is whether it captures something important in relation to the research question, and that there's no hard-and-fast proportion of the data a theme has to reach. What five out of twenty does change is how you write about it. You'd describe it as a pattern among some participants and spell out its boundaries, so readers can see who it applies to. **Sarah:** Let's try a small coding exercise, because I think it shows the difference between a group of codes and a theme. This is an invented excerpt, from a made-up participant, a man who moved across the country for work. "I moved here for the job, and the job's fine. It's the evenings. I'll text my sister back in Halifax, but she's hours ahead of me, so she's asleep by the time I'm home. I joined a hiking club, went twice. Everybody already had their people. I keep telling myself it takes a year." **Kiffer:** Let me code it line by line, staying close to the words. "I moved here for the job" gets relocation-for-work. "It's the evenings" gets evenings-hardest. The texting line gets contact-out-of-sync, because his sister is reachable but on a different clock. "Joined a hiking club, went twice" gets tried-a-group-then-stopped. "Everybody already had their people" I'd keep as an in vivo code, in quotation marks, because his phrase is sharper than anything I'd invent. And "it takes a year" gets waiting-it-out. **Sarah:** Then I'd group them two ways. One group is the coping codes, texting his sister and trying the hiking club. The other group pulls together everything about timing: the evenings, his sister being asleep, the club whose members had already found each other, and the year he's waiting out. **Kiffer:** So here's a question for everyone listening. Which of Sarah's two groupings is a category, and which one could become a theme? Pause the audio if you'd like more time. Take a few seconds. *(Pause)* **Sarah:** The coping group is a category. It gathers codes of the same kind under one heading, and coping strategies names an area without saying anything about it. The timing group could become a theme, because it can be stated as a claim. For people who move for work, loneliness shows up as a problem of timing, with the people they could reach living on another clock or already settled into groups that formed before they arrived. **Kiffer:** And notice where that theme's codes come from. Evenings-hardest describes an experience, contact-out-of-sync comes from a coping attempt, "already had their people" is a causal account, his explanation of why the club didn't work, and waiting-it-out is an interpretation. The theme draws on all four of the metacoding bins. Contact-out-of-sync even sits in both of your groupings, which is fine, because it lives in the coping branch of the codebook and still carries evidence for the theme. Categories usually sort codes by kind. A theme ties codes together by a shared idea, and that idea can draw on several categories at once. **Sarah:** With the obvious caveat that one invented excerpt can't support a theme. **Kiffer:** Right. This is a candidate. In the reviewing phase you'd test it on the rest of the transcripts and look hard for negative cases, like someone who moved for work and found a group in the first month. If those cases are common, the theme gets narrowed or split. **Sarah:** Question two. What makes a codebook entry usable by somebody who isn't you? I want to try one, because I suspect this is where I'd slip. My code is keeping-busy. Brief definition: the participant describes filling time with activity as a way of managing loneliness. Inclusion criteria: the participant describes an activity, and the activity is regular, something done most days or most weeks. Exclusion criteria: one-off events, like going to a single party. **Kiffer:** That's a reasonable first draft. Now let's test it on two borderline excerpts, both invented. The first: "Since Ruth died, I'm at the woodshop every day. The guys there don't talk much, which suits me." The second: "I've curled every Thursday night since I was nineteen. Wouldn't miss it." Using only the criteria Sarah just read, would a second coder apply keeping-busy to each excerpt? Take a few seconds. *(Pause)* **Sarah:** The woodshop one, yes. It's a daily activity, and it starts with Ruth's death. The curling one, no. He's done it since he was nineteen, it's about the sport and his Thursday night, and loneliness doesn't come into it. **Kiffer:** Those are the calls I'd make too, because we both know what the code is meant to capture. The question, though, was what a second coder would do with only your criteria. Read me your inclusion criteria again, and tell me which one the curling excerpt fails. **Sarah:** It's an activity. It's regular, every Thursday. Oh. It fails neither of them. The link to loneliness is in my brief definition, but I never wrote it into the criteria. A second coder working from the criteria would apply keeping-busy to the curling excerpt, I wouldn't, and we'd disagree for a reason that only exists in my head. **Kiffer:** That's a very common gap in a first draft. The brief definition carries the idea, the criteria leave part of it out, and the analyst fills the gap from memory without noticing. A second coder can't do that. So how would you fix it? **Sarah:** Add a third inclusion criterion: the participant links the activity to loneliness, to a loss, or to time that would otherwise feel empty, either in words or by clear inference within the passage. Add an exclusion: an activity described as a long-standing habit or for its own enjoyment, with no link to loneliness. And the curling excerpt becomes my negative example, because it's exactly the near miss a second coder needs to see. **Kiffer:** Good. There's one more test for the full definition, which is where the code sits next to its neighbours. If someone volunteers at a food bank to fill afternoons that feel empty, does that passage get keeping-busy, coping-volunteer, or both? **Sarah:** Both, since a passage can carry several codes. Coping-volunteer says what the activity is, and keeping-busy says what the activity is doing for the person. **Kiffer:** Then write exactly that into the full definition. Coders drift apart when two codes overlap and nobody has written down whether to apply one or both. Now the woodshop excerpt becomes the harder case under your new criterion. Does "since Ruth died" count as a link? **Sarah:** Since is one of the linguistic connectors, but here it might only be telling us when. Still, the daily routine begins with the loss, and "which suits me" tells us something about the kind of company he wants. I'd apply it. **Kiffer:** I'd make the same call, and I'd write it in the memo, something like, a routine the participant dates from a loss counts as a clear inference. Otherwise the next coder reads since as a date and leaves the passage uncoded. And there's one more consequence. You've changed the criteria, so every transcript you already coded with the old version needs a second look under the new one, with the change recorded in the audit trail. **Sarah:** Question three. What does an agreement statistic actually tell you? This is the harder example, so let's go slowly. Suppose two coders each read the same forty passages and decide, for each one, whether keeping-busy applies, yes or no. **Kiffer:** It's worth saying why we'd set it up that way. Because a passage can carry several codes, a common way to check agreement is code by code, with a yes or no from each coder for every passage. Now the invented results. Coder one says yes to three passages. Coder two says yes to two. Both of them say yes to one and the same passage. **Sarah:** Then four passages got at least one yes, three from coder one plus two from coder two, minus the one they share. That leaves thirty-six where both said no. Thirty-six plus the one shared yes is thirty-seven agreements out of forty, which is 92.5 percent. **Kiffer:** Which sounds excellent. So here's the question. Of those thirty-seven agreements, how many are agreements that the code is present? Take a few seconds. *(Pause)* **Sarah:** One. The other thirty-six are agreements that it's absent. So almost all of that 92.5 percent comes from two people agreeing that most passages aren't about keeping busy. **Kiffer:** And two coders would do that even if they barely understood the code. Kappa makes that visible. Coder one said no to thirty-seven of forty passages, which is 0.925, and coder two said no to thirty-eight, which is 0.95. If they were deciding independently, the chance they'd both say no is 0.925 times 0.95, about 0.879. The chance they'd both say yes is 0.075 times 0.05, which is under half a percent. Add those up and chance alone gives about 88.25 percent agreement. **Sarah:** So they beat chance by four and a quarter points, 92.5 compared with 88.25. And the most they could have beaten it by is the gap between 88.25 and a hundred, which is eleven and three quarter points. Four and a quarter divided by eleven and three quarters is about 0.36. **Kiffer:** That's kappa, about 0.36, which falls in the fair band. Krippendorff's alpha on the same decisions comes out almost the same, about 0.37. This pattern is often called the kappa paradox. In 1990, Alvan Feinstein and Domenic Cicchetti set it out in the Journal of Clinical Epidemiology, as the first of two paradoxes in a paper titled High Agreement but Low Kappa. When a category is rare, percent agreement can be high while kappa is low. **Sarah:** Okay, so here's what I'd expect a team to do next. They tighten the entry and recode, and in the next round both coders say yes to one passage, the same one, and no to everything else. Percent agreement is a hundred, and kappa is exactly one. So the problem looks solved. **Kiffer:** That's the trap in this example. Before you trust that perfect score, what would you want to know? Take a few seconds. *(Pause)* **Sarah:** How many times the code was applied. And the answer is once. The whole statistic rests on a single passage. **Kiffer:** Right, and it's fragile. If coder two had also said yes to one other passage, agreement would be thirty-nine out of forty, chance agreement would be about 92.75 percent, and kappa would drop to about 0.66. One passage moves the number from perfect agreement to the low end of substantial. For a code this rare, kappa tells you very little in either round. **Sarah:** So what would you do with a rare code? **Kiffer:** I'd do three things. First, report kappa or alpha code by code, with the number of times each code was applied, so a reader can see when a value rests on a handful of passages. Second, make sure the reliability sample includes enough passages where the code could plausibly apply, which may mean adding passages chosen for that purpose. Third, when a code is applied only a few times, have both coders talk through every application and record the decisions in the memo. A single kappa pooled across the whole codebook can hide all of this. **Sarah:** Question four, and this is one where we don't fully agree. Should a study like the one in this lesson report an agreement statistic at all? **Kiffer:** Let me hear your side of it first, Sarah. **Sarah:** My worry is what chasing the number does to the codes. The easiest codes to agree on are the most literal. A code for mentioning a body part would probably get a kappa near one. Somatic-absence asks whether a bodily image is carrying an absence, and that takes interpretation, so it will usually score lower. If a team has to reach 0.8, the pressure runs toward literal codes, and the interpretive ones get trimmed or dropped. **Kiffer:** That pressure is real, and it's a good reason to hold thresholds loosely. The statisticians who proposed the usual kappa bands described their own divisions as clearly arbitrary. Where I'd defend the statistic is in an applied team project, where several coders share one codebook across, say, twenty transcripts. If you report that a code appears in eleven of twenty transcripts, you want that count to reflect the participants, and to come out about the same whichever coder did each transcript. **Sarah:** But reviewers often read a kappa as a sign that the findings are valid, and it can't carry that weight. Two coders can agree perfectly on a code that misses the point of the interview. **Kiffer:** I agree, and that's the misuse I'd most want to avoid. Agreement tells you a codebook can be applied consistently. Whether the codes capture what matters is a separate question. My view is closer to a 2020 paper in the International Journal of Qualitative Methods, the journal of the University of Alberta's International Institute for Qualitative Methodology. Its authors argued that an agreement check can make coding more systematic and transparent, and can prompt reflexivity and dialogue within a team. **Sarah:** And Braun and Clarke make my case for reflexive thematic analysis. In a 2021 paper on quality in thematic analysis, they described demonstrating coding reliability as illogical, incoherent and ultimately meaningless in that approach, because meaning is understood as situated and the researcher's subjectivity is a resource for the analysis. **Kiffer:** For reflexive work, I agree with them. So our disagreement is narrower than it sounds. You'd leave the statistic out of more studies than I would. **Sarah:** That's fair. So where do we land? **Kiffer:** Here's a rule I think we can share. Decide which kind of thematic analysis you're doing before you decide on a statistic. If it's coding-reliability work, with several coders applying a fixed codebook, compute agreement code by code while the codebook is being developed, use it to find entries that need work, and report it with how often each code was applied. And call it what it is, a measure of how consistently the codebook was applied. **Sarah:** And if the analysis is reflexive, report how the interpretations were developed and discussed, and leave the kappa out. I can live with that. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, a pattern that shows up in many transcripts is a starting point. Check whether the interview guide put it there, and state a theme as a claim about the pattern. **Kiffer:** Second, the next coder works only from what you wrote down. Test every codebook entry on near misses, and make sure the criteria say everything the brief definition says. **Sarah:** And third, read every agreement statistic alongside how often the code was used. High percent agreement on a rare code mostly measures agreement about absence, and a perfect kappa can rest on a single passage. **Kiffer:** If you'd like more practice, take the keeping-busy entry from question two and write two more borderline excerpts of your own, one that should just qualify and one that should just miss. Then check whether the revised criteria sort them the way you intended. **Sarah:** Next time, it's Lesson six, Analysis Frameworks and Conceptual Models, where a preliminary codebook starts turning into a model that explains the patterns it found. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson six.