# Lesson 3: Sampling in Qualitative Research *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a made-up sentence of a kind you've probably read in a dozen qualitative papers. "Interviews continued until saturation was reached, at twelve participants." Kiffer, if you were reviewing that paper, would that sentence settle the question of sample size for you? **Kiffer:** It would open more questions than it settles. Which kind of saturation was it, codes or meaning? How was it judged, and by whom? And where were those twelve people drawn from? That same sentence can sit on top of a careful study, or on top of a sample that never reached the people the findings claim to describe. **Sarah:** And yet the number twelve has real data behind it. It's one of the most cited findings in qualitative methods. **Kiffer:** It does, and the study behind it is careful about what it found. The problems start when people cite the number without the conditions it came with. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. Where that number comes from, and what it can and can't carry, is a good way into Lesson three, Sampling in Qualitative Research, so let's start there. **Sarah:** Question one. When someone asks how many interviews are enough, why isn't twelve the answer, given that it comes from a real study? **Kiffer:** Let's start with the study itself. Greg Guest, Arwen Bunce and Laura Johnson published it in 2006. They had sixty in-depth interviews with women in Ghana and Nigeria, thirty in each country, collected for a study about social desirability and the accuracy of self-reports in reproductive health research. The participants were sex workers, all women at high risk of HIV. The team coded the Ghanaian interviews first, six at a time, and tracked when each new code appeared. **Sarah:** And by the twelfth interview, ninety-two percent of the codes had turned up. **Kiffer:** Right. So here's a question for everyone. That ninety-two percent is a share of some total. What total is it a share of? Take a few seconds. Pause the audio if you'd like more time. *(Pause)* **Kiffer:** It's a share of all the codes they eventually found across the thirty Ghanaian interviews, which came to one hundred nine. One hundred of those had appeared by interview twelve. So the figure could only be calculated after all thirty were coded. A team that stops at twelve has no total to divide by. Every code they have was found by interview twelve, so the same calculation would always give them one hundred percent. **Sarah:** So the twelve comes from looking backward at a finished dataset. And when they added the Nigerian interviews? **Kiffer:** Only five new codes, for a final total of one hundred fourteen. That sounds like strong support for twelve, until you look at what kind of sample it was. The authors described it as a relatively homogeneous population with fairly narrow objectives, and they wrote that twelve interviews will likely not be enough if a group is relatively heterogeneous. **Sarah:** Which is why the number carries less weight for something like the course's loneliness interviews, which run from age eighteen to eighty-two and were chosen to differ from each other. **Kiffer:** Yes. And there's a second gap. What Guest and colleagues counted was codes. Monique Hennink, Bonnie Kaiser and Vincent Marconi later separated codes from meaning. In their twenty-five interviews, code saturation came at nine, while a richly textured understanding of what the codes meant took sixteen to twenty-four. Their shorthand is that code saturation tells you when you've heard it all, and meaning saturation tells you when you understand it all. **Sarah:** And Hennink and Kaiser's later review of empirical tests found the usual range was nine to seventeen interviews, mostly for code saturation, and mostly in fairly homogeneous samples with narrowly defined aims. So the honest answer to "how many?" starts by asking what kind of saturation you need, and in what kind of sample. **Kiffer:** That's the answer I'd give. A number from this literature is a reasonable planning estimate when your study looks like the studies that produced it. It's weak evidence once your sample is broad, your question is open, or you're after meaning more than a list of codes. **Sarah:** Question two. Here's an invented study, so treat every detail as made up. A team interviews fourteen adults aged seventy and over, all recruited through programs at two seniors' centres in a mid-sized city. Their conclusion reads, "For older adults, loneliness is primarily a consequence of losing a spouse." I want to try judging this one myself. **Kiffer:** Go for it, but let's give everyone a moment first. Can this sample support that conclusion? Take a few seconds. *(Pause)* **Sarah:** Okay. My first reaction is that fourteen is far too few. If you want to say what loneliness is primarily caused by, you need numbers, maybe a few hundred people. So the fix is a much bigger sample. **Kiffer:** Let's test that. Picture the same team interviewing four hundred people, all from those same two seniors' centres. Is the conclusion safe now? **Sarah:** Well, no, it isn't. It's still only people who come to seniors' centres. People who are housebound, or living in long-term care, or who just don't go to group programs can't be in it at any size. And the people who are there are getting out of the house, which already says something about their social lives. **Kiffer:** So what does that mean for the finding about losing a spouse? **Sarah:** That it could partly reflect where they recruited. If someone's loneliness comes from hearing loss, or money, or not being able to drive, those same things might keep them away from the centre. So the routes into loneliness that keep people at home are the ones least likely to show up, and a bigger sample from the same centres would have the same gap. **Kiffer:** That's a check worth doing every time. Ask whether a bigger version of the same sample would fix the problem. If it wouldn't, the problem is who's in it. And there's a second issue, hiding in one word of the conclusion. **Sarah:** "Primarily." That's a frequency claim. It says most older adults' loneliness has this cause, and a nonprobability sample can't support a claim about how common something is, whether it has fourteen people or four hundred. **Kiffer:** Right. So try rewriting the conclusion so the sample can carry it, and then tell me how you'd change the design. **Sarah:** The conclusion could say that among older adults who attend seniors' centre programs, losing a spouse was one route into loneliness, and then describe how that route worked for them. For the design, I'd add recruitment channels that reach people who don't attend, like home-care agencies, meal delivery programs, family physicians and long-term care homes. And I'd write down who still isn't covered. **Kiffer:** Good. And if the real question is about bereavement, a homogeneous sample of recently widowed people, recruited through several of those channels, would answer it more directly. There's a real-world parallel here. Even large surveys keep a list of who they miss. Statistics Canada's Canadian Community Health Survey leaves out several groups, including people living on reserves, full-time members of the Canadian Forces, and people living in institutions, which includes long-term care. Statistics Canada puts all of its exclusions together at under three percent of adults. **Sarah:** For a loneliness question, that three percent could include many of the people you'd most want to hear from. **Kiffer:** Which is one place a qualitative study adds something a survey can't. A survey tells you how common loneliness is. Statistics Canada's Canadian Social Survey found in 2021 that thirteen percent of people aged fifteen and older always or often felt lonely, and that survey also left out people living in institutions. A study that deliberately goes into long-term care homes can tell you what loneliness is like for people the survey never reached, and how it comes about. **Sarah:** Question three. What does a flat saturation curve actually show? **Kiffer:** Let's work through an invented example, and I'll give the numbers slowly. A team is studying loneliness among family caregivers of people living with dementia. Their first ten interviews come from a caregiver support group at an urban hospital. They code in the order they interviewed and count new codes. The first interview gives eleven new codes, the second six, then four, three and two, and one each in the sixth and seventh. Interviews eight, nine and ten add nothing new. **Sarah:** So twenty-eight codes in all, and three interviews in a row with nothing new. **Kiffer:** Here's the question. If the team stopped there and wrote that code saturation was reached, would that be a fair claim? Take a few seconds. *(Pause)* **Sarah:** I'd say it's fair for the people they actually interviewed. Among caregivers who come to a hospital support group, the list of codes has stopped growing. As a claim about caregivers in general, it goes too far, because every one of them came through the same door. **Kiffer:** That's the right line to draw. Now suppose the team keeps going, and the next six interviews come from home-care offices in two rural communities. Interview eleven adds four new codes, twelve adds two, thirteen and fourteen add one each, and fifteen and sixteen add nothing. The new codes are things like the long drive to the nearest respite program, and being known by everyone in town. **Sarah:** So eight new codes from the rural interviews, and thirty-six in total. Let me do the backward calculation the way Guest and colleagues did. By interview six, the team had twenty-seven codes. Out of the twenty-eight they had at interview ten, that's ninety-six percent, which beats the famous figure. **Kiffer:** And what does it look like when you divide by the final total? **Sarah:** Twenty-seven divided by thirty-six is seventy-five percent. And the rural interviews supplied eight of the thirty-six, which is about twenty-two percent of the final codebook. So the flat stretch from eight to ten was real, and what it measured was how alike the support-group caregivers were. **Kiffer:** There's a published stopping rule that makes this sharper. Jill Francis and colleagues proposed one in 2010 for theory-based interview studies. Analyse an initial ten interviews, then stop once three more in a row add no new ideas. If you apply just those two numbers, what would the rule have done here? **Sarah:** It would have kept going past ten and found the rural codes, but only because the next interviews happened to be rural. If the team had recruited three more people from the same support group, those three might well have added nothing, and the rule would have said stop. **Kiffer:** So the numbers in a stopping rule can only see the interviews you give them. They can tell you when a recruitment channel has run dry, and they have no way of telling you about channels you never opened. Francis and colleagues saw this coming. They said the first ten should already be spread across the groups that matter for the study, with rurality as one of their own examples, because a sample that's too alike can reach saturation too early. That's the trap in this example, and it's why saturation and sampling strategy have to be read together. **Sarah:** So how would you set the study up so the curve means something? **Kiffer:** I'd treat the recruitment channel as a dimension of variation from the start, the way you'd set a quota, and judge saturation within each cell. It also helps to code in an order that alternates between the cells, so a run of interviews with nothing new can't come from one cell alone. And the methods section still names who isn't covered, here caregivers who aren't connected to any service at all. **Sarah:** And the code count is only half the story. The rural interviews could also change what an existing code means. **Kiffer:** That's meaning saturation again. Say the urban group produced a code about having no one to hand off to. In the urban interviews it was about plans falling through at short notice, or a sibling who works full time. In the rural ones it's about the only person who could step in being a neighbour who's getting older too. That code still counts once, and what it means has changed. **Sarah:** Question four. Does it matter what you call your sampling strategy? Here's an invented methods paragraph. "Participants were selected through theoretical sampling until theoretical saturation was reached. Twenty-four adults were recruited through flyers at three community health centres, with targets of twelve women and twelve men set before recruitment began." **Kiffer:** If you were reviewing that paragraph, what would you ask the authors for? **Sarah:** I'd ask what the rounds were. Theoretical sampling means the analysis of early interviews decides who you look for next. Here the targets were fixed in advance and filled from flyers, which is quota sampling, with convenience recruitment inside each cell. So either they describe the rounds, what they learned and who they went looking for because of it, or they relabel the design. **Kiffer:** And what about the claim of theoretical saturation in that same sentence? **Sarah:** That's the most demanding kind of saturation, and it depends on that iterative sampling. With one recruitment phase, they could still claim code saturation, or meaning saturation, as long as they show how they assessed it. **Kiffer:** The label matters because it tells a reader what the design can carry. Calling a sample theoretical implies the developing theory was tested on cases chosen to strain it, including disconfirming ones. If that never happened, the paper is claiming a kind of testing it didn't do. **Sarah:** Network sampling has its own version of this. Here's another invented study. A snowball sample on loneliness among older gay men who aren't out to their families. There are three seeds. One comes from a social group for older gay men, one from a dating app, and one was referred by a family doctor. There are thirty participants in the end, and twenty-one of them trace back to the social group seed. What does that composition mean for the findings? Take a few seconds. *(Pause)* **Kiffer:** Seventy percent of the sample comes from one network, and it's a network built around getting together. So the accounts will lean toward men who have at least one place to belong, and any finding about what helps may give too much weight to group belonging. The men who are most isolated are the least likely to be reached by any chain, because a chain needs someone to pass the invitation along. **Sarah:** So what would you report, and what would you change? **Kiffer:** I'd report the chain structure plainly, how many participants came through each seed, and compare accounts across the chains. And next time I'd choose more seeds, deliberately, so the chains start in different places. That's purposive thinking applied to the seeds. **Sarah:** A large respondent-driven study in Canada chose its seeds that way. The Engage study recruited gay, bisexual and other men who have sex with men in Montreal, Toronto and Vancouver between 2017 and 2019, two thousand four hundred forty-nine participants in all. Seeds were purposively selected with help from community committees, each participant got six coupons, and people were paid fifty dollars for their visit and fifteen dollars for each peer they recruited. **Kiffer:** And the analysis then gives each person less weight the bigger their reported network is, because people with bigger networks are more likely to be recruited. That's the inference machinery from the lesson, and it rests on assumptions about how people recruit within their networks that are still debated. Even so, the whole design begins with a purposive choice of who the seeds are. **Sarah:** Okay. Now the part where we disagree. I'm not convinced saturation should be the standard for qualitative work at all. **Kiffer:** Make the case, and I'll tell you where I think it goes too far. **Sarah:** Virginia Braun and Victoria Clarke put it well. Saturation assumes there's a fixed stock of meaning sitting in the data, and you're finished when you've taken inventory. In reflexive thematic analysis, meaning is generated by the analyst's interpretation, so there's always more that could be made. "Saturation was reached" sounds objective, and it hides a judgment call. And my own worry is that in practice it often works as an alibi for stopping where the team was going to stop anyway. **Kiffer:** I agree it gets misused, and for reflexive or constructivist work I think your argument holds. Where I'd push back is in applied public health. An ethics board, a funder or a reviewer needs some way to judge whether a team stopped too early. Code saturation, assessed openly and reported with the curve, gives them something they can check. Information power is a good planning tool, and it gives no number at all. **Sarah:** But a checkable number can still be the wrong thing to check. Our caregiver example had a perfectly flat run, and it was flat because of who was recruited. **Kiffer:** That's fair, and it's the strongest version of your argument. A saturation curve can only reflect the sample that produced it. I'd still keep saturation in the toolkit for applied studies, as long as it's reported with its conditions attached: which kind, how it was assessed, and within which part of the sample. **Sarah:** I'm more skeptical than Kiffer, especially for interpretive work, but we agree on the practical rule. Plan the sample with information power, say which standard you're using and how you assessed it, and whatever standard you use, name the variation the sample didn't capture. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, a number like twelve is a finding from particular studies, mostly about code saturation in fairly homogeneous samples with narrow aims. Before you borrow it, check whether your study looks like those. **Kiffer:** Second, where you recruit shapes which forms of a phenomenon can show up at all. A bigger version of the same sample won't fix that, and a flat curve inside one recruitment channel tells you nothing about the channels you never used. **Sarah:** And third, match the label and the claim to the design. Purposive, quota, theoretical and snowball samples each support different claims, and a word like "primarily" needs a probability sample behind it. **Kiffer:** If you'd like more practice, take the seniors' centre study from this episode. Rewrite its conclusion so the sample can carry it, then write the one sentence that names the variation the sample didn't capture. **Sarah:** Next time, it's Lesson four, Qualitative Data Collection, where we move from who's in the data to how interviews, focus groups and field notes actually get made. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson four.