# Lesson 4: Qualitative Data Collection *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a finding from a large hospital in Toronto. Researchers used an electronic system that logged every use of the soap and hand rub dispensers on two transplant units. When a hand hygiene auditor was standing in view, the nearby hallway dispensers were used roughly three times as often as they were otherwise. **Kiffer:** Which is awkward, because watching people is supposed to be the remedy for unreliable self-report. Clinicians overestimate how often they clean their hands, so you send in an observer, and then the observer changes the very behaviour they came to measure. **Sarah:** So asking is biased, and watching is biased. What's left for a qualitative researcher to collect? **Kiffer:** Quite a lot, once you're clear about what each kind of data is evidence of. The dispenser log, the auditor's tally and an interview with a nurse each answer a different question. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. Knowing what each kind of data can show is the thread running through Lesson four, Qualitative Data Collection, so let's start there. **Sarah:** Question one. Is watching more trustworthy than asking? In the lesson's hand hygiene example, observation beats self-report, and the Toronto study complicates that. **Kiffer:** The Toronto study was published in 2014, and the authors drew a pointed conclusion. Many publicly reported hand hygiene rates come from audits by observers, so those rates may be substantially inflated. That's the Hawthorne effect, direct observation's own version of reactivity. And look at what caught it. The dispenser log ran for months whether or not an auditor was there. It's a behaviour trace, the kind of indirect measure Webb and his colleagues described in 1966. **Sarah:** With the usual limit of a trace. The log couldn't say who used each dispenser, or how many chances to clean hands there were. It could only show that use went up when the auditor arrived. **Kiffer:** Right, and that was enough for the question they were asking. The same logic tells you what interviews are good for. Kvale and Brinkmann describe two pictures of the interviewer. One is a miner, digging out facts and feelings that are already sitting there. The other is a traveller who explores the topic alongside the participant and comes back with an account that grew out of their conversation. **Sarah:** And if you think like a miner, every gap between what people say and what they do looks like a failure of the interview. If you think like a traveller, the account is the thing you came for. When Maya says the SkyTrain at nine on a Sunday night is what loneliness brings to mind, the claim the interview supports best is about what that ride means to her. How often she rides it is a question a transit log would answer better. **Kiffer:** Let's put that thinking to work on observation with a made-up example. Picture a team doing spot observation in the lounge of a long-term care home. At random moments between nine in the morning and five in the afternoon, an observer walks through the lounge and records what each resident there is doing. Over two weeks that adds up to four hundred resident moments. In two hundred twenty of them, the resident is sitting alone, not talking with anyone and not doing any visible activity. **Sarah:** Two hundred twenty divided by four hundred is fifty-five percent. So residents in this home spend a little over half their time alone and idle. **Kiffer:** That's the arithmetic. Here's the question for everyone. What can that fifty-five percent tell you about loneliness in this home? Take a few seconds. Pause the audio if you'd like more time. *(Pause)* **Sarah:** I overstated it a moment ago. The fifty-five percent describes residents who were in the lounge, during the day. Anyone who stayed in their room was never counted, and residents who withdraw to their rooms may be exactly the people we most want to know about. Evenings weren't sampled at all. **Kiffer:** That's the sampling frame of the observation, and it's easy to miss because the number looks so clean. What else would you flag? **Sarah:** Being alone and feeling lonely are different things. Spot observation records what people are doing. Some of those residents might enjoy a quiet hour, and someone in a crowded room can still be lonely, which is Maya's whole point about the train. **Kiffer:** And the residents can see an observer walking through, so the Hawthorne effect applies here too. The defensible statement is narrow. During daytime hours, residents in the lounge were alone and inactive in about half of the observed moments. That tells you where to look, and interviews tell you what the time alone is like. **Sarah:** Question two. Where does description stop and interpretation start? This comes up with field notes, and it comes up with transcripts. **Kiffer:** Let's stay in the care home. Here's an invented sentence from someone's expanded notes after lunch. Mr. D. sat alone at the corner table again, looking lonely and withdrawn. He pushed his food around because he had no one to talk to. Which parts of that would a camera have caught? Take a few seconds. *(Pause)* **Sarah:** A camera catches that he sat at the corner table, that nobody else sat there, and that he moved his food around the plate. It doesn't catch the words lonely or withdrawn, and it certainly doesn't catch the word because, which carries a causal claim. The word again is in between. It counts as description only if the earlier lunches are in the notes too. **Kiffer:** Good. A disciplined version might read like this. At twelve ten, Mr. D. sits at the corner table, his back to the window. Three empty chairs. He moves food around the plate and eats about half. A staff member asks, all done? He nods. Then, in a separate dated memo: third lunch in four days that Mr. D. has eaten alone. Loneliness, preference, or trouble hearing in a noisy room? Ask staff about the seating plan. **Sarah:** Trouble hearing is a good example. Withdrawn might describe a man who can't follow conversation over the clatter of dishes. If that interpretation had gone into the description, it would have hardened into a fact, and nobody could challenge it later. **Kiffer:** Transcripts carry the same risk, because the convention decides what counts as the record. Here's an invented verbatim excerpt from an older man. I don't, um, I don't really, I don't go out much now. Not since. Then a bracketed pause. Me and Doug used to go to the Legion every Friday. Then a bracketed laugh, and, anyway. Sarah, what does intelligent verbatim do to that? **Sarah:** It takes out the um and the false starts and keeps his own wording and grammar. So it begins, I don't go out much now. Not since. **Kiffer:** And what has the analyst lost in that version? **Sarah:** The phrase, I don't really. He started one sentence, dropped it, and started a safer one. Maybe he was heading toward something like, I don't really have anyone. We can't know that, and the transcript shouldn't pretend to. But the verbatim version lets an analyst notice the restart and wonder about it. **Kiffer:** And clean verbatim would go one step further and change his, me and Doug, to the more formal, Doug and I, which edits out a little of how he talks. Each convention is legitimate for some studies, and each one decides in advance what a later analysis is able to notice. **Sarah:** A lot of first drafts now come from automatic transcription. Which convention does that give you? **Kiffer:** Whichever one the software's designers chose. Whisper, a widely used open speech recognition model, drops many filler words and repetitions, according to a 2024 paper from a team that built a more verbatim version of it. So you can believe you have verbatim when you have something close to intelligent verbatim. Another team tested Whisper in 2023 and reported in 2024 that roughly one percent of its transcriptions contained whole phrases nobody had said. Those invented phrases turned up more often for speakers with aphasia, and in recordings with long stretches of silence. **Sarah:** And long silences are where loneliness interviews do some of their most important work. I assume the software has been updated since then? **Kiffer:** It has. When the same team retested late in 2023, most of the recordings that had produced invented phrases no longer did, though some still did. Treat the finding as a reason to check whatever tool you use. I'd write down the convention you want before you start, listen to every minute of audio while you correct the draft, and if the tool sends recordings to an outside service, make sure your ethics approval covers that. That listening also counts as familiarization, so the time is well spent. **Sarah:** Question three. What makes a probe leading? I think this is harder than it sounds, because in the moment a leading probe often feels like empathy. **Kiffer:** Let's try one. Here's an invented excerpt from a hypothetical interview with a woman in her forties who moved to Kamloops for work last year. The interviewer asks what a typical weekend looks like. She says, weekends are, um, quiet. I do the groceries, I clean. My sister calls on Sundays. It's fine. It's, yeah. It's fine. What probe would you use next? Sarah will try it aloud, but everyone take a few seconds to choose your own first. *(Pause)* **Sarah:** I'd go after the it's fine, because she said it twice and it doesn't sound fine. So I'd say, it sounds like weekends can be pretty lonely for you, is that right? **Kiffer:** Check that probe with one quick test. Does it contain a word or an idea the participant hasn't used yet? **Sarah:** Lonely. She never said lonely. I put my category into her mouth and asked her to agree with it, which makes it a leading probe. She can say yes, or she can push back on my word, and either way she's now talking inside my frame. In a loneliness study it's worse, because she knows what the study is about and may feel she ought to confirm it. **Kiffer:** There's a social cost as well, because saying no to a kind-sounding interviewer is hard. So what would you use? **Sarah:** A silent probe first. Just wait after the second it's fine. If nothing comes, an echo probe, saying, it's fine, back to her with a slight rise. Or a tell-me-more probe about something she offered herself, like, tell me about the Sunday calls. **Kiffer:** Each of those keeps her words in charge. I'd be a little careful with the echo here, because on a repeated it's fine it can sound like you doubt her, so tone matters, and silence is often gentler. And she's entitled to leave it at fine. If she does, that's data too, and you move on, the way the interviewer let it rest when Amira said that to explain everything is to live through it again. **Sarah:** Question four. If someone else collected the data, how far can you trust a pattern you find in it? This one matters here, because working with the course's loneliness interviews is a form of secondary analysis. **Kiffer:** Here's a harder made-up case. A researcher finds a deposited dataset of twenty-four semi-structured interviews with family caregivers of people living with dementia. The original study was about respite services. She wants to write about caregiver loneliness, so she reads all twenty-four and marks every passage where a caregiver ties something to feeling alone or cut off. She reports that the cost of care is the dominant source of loneliness, tied to it in nineteen of the twenty-four interviews. Grief for the person they used to know is tied to it in eleven. **Sarah:** Nineteen of twenty-four is about seventy-nine percent. That sounds like a strong pattern. **Kiffer:** Before anyone accepts it, what would you check first, given that this is somebody else's dataset? Take a few seconds. *(Pause)* **Sarah:** The original interview guide. A study about respite services probably asked about money directly. **Kiffer:** In this invented case it did. One domain of the guide asks, how do the costs of care affect you? So she goes back and marks where each mention of cost first appears. In sixteen of the nineteen interviews, cost first comes up in answer to that question. In the other three, the caregiver raised it before the question was asked. **Sarah:** So unprompted, cost comes up in three of twenty-four, which is one in eight. What about grief? **Kiffer:** The guide never asks about it anywhere. All eleven mentions were offered unprompted. **Sarah:** Then the picture flips. The topic that looked dominant was mostly produced by the guide, and the one that looked secondary came up spontaneously in eleven of twenty-four interviews, about forty-six percent, from people who were never asked. **Kiffer:** That's the trap, and there's a second one underneath it. The sixteen prompted answers about cost may still be rich and painful, so prompted doesn't mean unimportant. And check where the cost question sat in the guide. If it came early, people had little chance to raise cost on their own first, so three of twenty-four may understate how often people would have raised it unprompted. A count can't settle how much something matters in either direction. **Sarah:** And the guide never asked about friends or faith communities, so if those barely come up, that may say more about the guide than about people's lives. So how should she write it up? **Kiffer:** Report prompted and unprompted mentions separately, describe what people said about cost and about grief in their own terms, and quote the relevant guide questions in the methods. She also needs to check consent. In Canada, the Tri-Council Policy Statement on research ethics allows secondary use of non-identifiable information with research ethics board review and no new consent. Using identifiable information without consent has to meet a set of conditions, including that seeking consent is impossible or impracticable. And I'd add that detailed caregiving stories can be harder to make truly non-identifiable than people expect. **Sarah:** And the course's loneliness interviews have the same feature. Amira's answer about wahda came from a probe asking whether loneliness was even her word. So when a pattern shows up across the transcripts, it's worth checking which question or probe came just before it. **Kiffer:** That leaves the question where Sarah and I don't fully agree, which is whether focus groups have any place in loneliness research. **Sarah:** I'm skeptical. Loneliness carries stigma, and stigmatizing topics are on the lesson's list of cases where focus groups are the wrong choice. Maya's line that loneliness feels like admitting she's failing at being twenty-two is exactly the kind of thing few people would say in a room of strangers. And the person who does disclose can't control where the story goes afterward, because confidentiality among participants can't be guaranteed. **Kiffer:** Those are real risks. The same lesson also lists populations who are reluctant to speak alone, and Jenny Kitzinger made the point in 1995 that groups can encourage participation from people who don't want to be interviewed on their own. Public health researchers who ran focus groups on sensitive health topics reported in 2000 that the method could draw out views on those topics, and that the group's discomfort and disagreement were useful data in their own right. **Sarah:** I'd add a recruitment problem, though. The loneliest people may be the least likely to come to a group session at all. So a focus group on loneliness may hear mostly from people who are coping well enough to show up. **Kiffer:** That's a fair point, and it's a sampling problem the design has to name. Where I'd still use groups is for the public side of loneliness. Picture a group of new parents, or newcomers to a city, where one person says, I haven't really talked to another adult in days, and three others say, me too. You learn what's sayable, which words people share, and where the conversation stops. That's useful material for anyone designing a public health campaign. **Sarah:** Or you learn what the first speaker made it safe to say, and everyone else agrees. A group can produce agreement as easily as it reveals it, and the person who disagrees often goes quiet. **Kiffer:** I agree that's the core risk, and it's why moderating is such a demanding skill. So here's where I land. I'd use groups for the shared vocabulary of loneliness, with peers who aren't in power relationships with each other. I'd ask about people like us, or about a short invented vignette, so nobody has to disclose anything personal, and I'd offer an individual interview to anyone who wants to say more. **Sarah:** I'm still more cautious than you are, but we agree on the practical rule. Decide how much personal disclosure the question needs. If it needs one person's private account, use an interview. If it needs the public language, a carefully composed group can work, as long as nobody has to disclose anything to take part and the write-up says who didn't come. **Kiffer:** Let's pull it together with three things to take away. **Sarah:** First, every kind of data is evidence of something specific. Interviews give you accounts and meaning, observation gives you behaviour under observation, and traces give you behaviour without its meaning. Pairing them is often the strongest design. **Kiffer:** Second, keep description and interpretation in separate places, in field notes and in transcripts. The transcription convention, including a software default, decides what later analysis can notice, so choose it on purpose and write it down. **Sarah:** And third, data carry the marks of how they were collected. A probe can put a word in someone's mouth, and a guide can put a theme into a dataset. Before you count a pattern, check where it came from. **Kiffer:** If you'd like more practice, take the caregiver problem from this episode and try it on the course's loneliness interviews. Pick a pattern you noticed while getting familiar with the transcripts, and for each mention, mark whether a guide question or probe prompted it or whether the participant raised it unprompted. **Sarah:** Next time, it's Lesson five, Finding Themes and Building Codebooks, where that familiarization reading becomes the raw material for coding, and for building codes into themes. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson five.