# Lesson 2: Research Questions, Theory and the Literature *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's something a lot of people say when they start their first qualitative project. "My study is exploratory, so I'm going in with a blank slate. No theory, no hypotheses, and I've kept my reading light so the data can speak for themselves." **Kiffer:** It sounds careful, and it comes from a good instinct. The trouble is that nobody reaches a transcript with a blank slate. Somebody decided loneliness was worth studying, wrote the interview guide, and chose who to ask. Theory went into every one of those choices before the first interview started. **Sarah:** So the data never really speak for themselves? **Kiffer:** The data answer the questions you put to them, and you're the one who chooses the questions. That's why it pays to know which theory you're bringing and how tightly you're holding it. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. That question about the theory you bring with you sits right at the centre of Lesson two, Research Questions, Theory and the Literature, so let's start there. **Sarah:** Question one. If my study is exploratory, should I stay away from theory and the literature until I've analysed my data? **Kiffer:** There's a real history behind that instinct. In 1967, Barney Glaser and Anselm Strauss, the sociologists who developed grounded theory, advised researchers, quote, "literally to ignore the literature of theory and fact on the area under study." Their worry was contamination. They wanted their categories to come from the data, without existing concepts shaping them in advance. **Sarah:** And that advice has been argued over ever since. Charmaz, whose constructivist version of grounded theory comes up later in the course, holds that researchers can't keep their own influence out of an analysis. She still suggests letting the formal literature review wait until later in the study, so your own ideas get room to develop first. **Kiffer:** Right, so even inside grounded theory, the live debate is about when and how you use the literature. That's where sensitizing concepts come in. The term comes from the sociologist Herbert Blumer, in a 1954 paper. He contrasted definitive concepts, which, in his words, "provide prescriptions of what to see," with sensitizing concepts, which "merely suggest directions along which to look." Stigma, social capital, and the difference between loneliness and being alone can all work that way. **Sarah:** So exploratory and confirmatory work both use theory. What separates them is how tightly the theory constrains the answer. **Kiffer:** That's the distinction from the lesson, and it's worth testing on a pair of questions. A sensitizing concept points you somewhere and leaves open what you'll find there. A hypothesis tells you what you should find. **Sarah:** Then here are two research questions about the course's loneliness interviews. The first asks, "How, if at all, does stigma figure in the way participants talk about seeking help for loneliness?" The second says, "Participants who describe their loneliness as shameful will describe delaying or avoiding help." So here's a question for everyone listening. In which of those two is stigma working as a sensitizing concept, and in which is it part of a hypothesis? Take a few seconds. Pause the audio if you'd like more time. *(Pause)* **Kiffer:** In the first question, stigma is a sensitizing concept. It tells you where to look, and the words "if at all" leave room for the answer that stigma barely matters. That's an exploratory posture, even though theory clearly informs it. In the second, stigma sits inside a hypothesis that predicts a pattern, so the posture is confirmatory. **Sarah:** Can the second one still be studied with twenty transcripts? **Kiffer:** It can, as long as it's tested the way confirmatory qualitative work tests things. You look for cases that fit, and you look especially hard for cases that break the pattern, such as a participant who describes deep shame and still asked for help right away. What you can't do is turn it into a rate. A sentence like "seventy percent of ashamed participants delayed help" claims more than a purposive sample can support. **Sarah:** Let me push on the practical side. If I've read the work by Cacioppo and colleagues on loneliness and being alone, won't I just find their distinction everywhere? **Kiffer:** That's a fair worry. What I'd suggest is writing a short memo before you code, listing the concepts you're carrying and what you expect to see. Later, compare your codes with that memo. Where the findings match your expectations exactly, look harder for the cases that don't fit. The memo makes your assumptions visible to you, and to a reader if you report it. **Sarah:** Question two. What does a paradigm statement actually commit you to? I've read methods sections that declare a paradigm in one sentence and never mention it again. **Kiffer:** Then let's appraise one. This is a made-up methods paragraph, written to contain several problems. "This study adopted a constructivist paradigm and used Charmaz's constructivist grounded theory. Twenty adults were each interviewed once with a semi-structured guide covering definitions of loneliness, coping and stigma. Two coders independently coded all transcripts and reached a kappa of 0.82, confirming that the themes were accurate. Three themes emerged from the data. The most common, stigma, appeared in fifteen of the twenty interviews, showing that stigma is widespread among lonely adults." Which sentence fits the declared paradigm least well? Take a few seconds. *(Pause)* **Sarah:** I'd pick the kappa sentence. A constructivist study treats meaning as built between participants and researchers, so there's no single correct coding sitting in the transcripts for two coders to converge on. A kappa of 0.82 tells you that two people applied a codebook in similar ways. The word "confirming" treats their agreement as proof that the themes are accurate. Agreement between coders can't prove accuracy in any paradigm, and in a constructivist study the idea of one accurate coding doesn't fit at all. Does that mean a constructivist team should never code in pairs? **Kiffer:** Some interpretive teams do code in pairs, and I think that's reasonable when the purpose is to surface different readings and talk them through. Then you report the discussion and what changed because of it. A postpositivist study could report kappa as one credibility procedure among several, and that would fit its stated assumptions. The trouble in this paragraph is the mismatch between the stance and the claim. **Sarah:** I'd flag "themes emerged" as well. In a constructivist study, the researchers built those themes. Saying they emerged hides the very role the paradigm says the researchers played. **Kiffer:** Agreed. And the last sentence has two problems. Stigma was a topic in the guide, so its showing up in fifteen interviews partly reflects the fact that the interviewer asked about it. **Sarah:** And "widespread among lonely adults" is a claim about prevalence, made from twenty interviews. **Kiffer:** Right. Fifteen of twenty is a fair description of this dataset, and the leap to lonely adults in general is where it goes wrong. There's one more issue, and it's easy to miss. Grounded theory depends on theoretical sampling, which means choosing later participants to develop the categories you're building. A single round of interviews with a fixed guide looks more like qualitative description or thematic analysis. **Sarah:** So what would a reviewer ask for? **Kiffer:** Start with the coding. Either keep the constructivist stance and describe how the coders' different readings were discussed and recorded, or, if the team actually worked from postpositivist assumptions, say so and say what the kappa does and doesn't show. **Sarah:** Then for the last sentence, describe stigma by the forms it took across the interviews, say plainly that the guide asked about it, and drop the claim about lonely adults in general. That leaves the design. **Kiffer:** For the design, the authors either explain how theoretical sampling happened, or they name the design their data actually support, which here is probably qualitative description. **Sarah:** Honestly, this example makes me want to defend pragmatism. Most of that paragraph's trouble came from a label the authors didn't need. A lot of qualitative work in public health is applied. People want to improve a program or design a service, and the interest holders waiting for the findings have little use for a page of philosophy. Why not say, "We chose methods to fit the question," and leave it there? **Kiffer:** Pragmatism is a recognized position, the usual stance in mixed-methods research and common in implementation research, so I'm sympathetic. Where I'd push back is on leaving the philosophy out entirely. "What works" still needs a definition. Works for which purpose, and judged by whom? And every analysis treats participants' accounts as something, for example as windows onto an experience or as the experience as it's told. A reader needs to know which before judging a kappa or a count. **Sarah:** But declaring a paradigm didn't help our invented authors. They named one and contradicted it three sentences later. The label can turn into a ritual sentence that reviewers expect and nobody reads closely. **Kiffer:** I agree with that, and it's the usual criticism of pragmatism turned around. Critics worry that pragmatism can become cover for leaving your assumptions undisclosed. A paradigm label with nothing behind it can hide assumptions just as well. **Sarah:** So where do you land? **Kiffer:** I'd keep the label, because it tells a reviewer which standards to apply. I'd put more weight on three plain questions answered in writing. What do you treat the accounts as? What would count as a finding? And what role did you play in producing it? If you mix approaches, say which assumption governs which part of the analysis. **Sarah:** I can live with that, as long as a pragmatist can answer those same three questions and call it done. **Kiffer:** Agreed. So here's the rule we share. Whatever you call your stance, answer the three questions in the methods section, and make sure your coding, counting and claims stay consistent with your answers. **Sarah:** Question three moves to the literature. Why can't you summarize a body of qualitative evidence by counting how many studies found something? **Kiffer:** Counting has a place, and one synthesis method is built on it. Let's start one step earlier, though, with finding the studies. SPIDER, the search frame from the lesson, was designed for qualitative questions. In 2014, a research team tested it. They compared SPIDER with PICO, and with a version of PICO that added qualitative design terms, on a review of qualitative studies about the health care experiences of people living with multiple sclerosis, in three databases. **Sarah:** And SPIDER came out ahead? **Kiffer:** On one measure. It was the most specific in every database, which the authors used to mean that a larger share of what it found was relevant. It was also less sensitive than PICO, so it found fewer of the relevant studies. Across the three databases, PICO returned more than twenty-three thousand records and turned up eighteen relevant studies. SPIDER returned two hundred thirty-nine records and turned up thirteen. **Sarah:** So SPIDER would have saved the team from screening more than twenty-three thousand records, at the cost of five relevant studies. What did the authors recommend? **Kiffer:** PICO for a fully comprehensive search, and the version with qualitative terms when time and resources are limited. Because of the risk of missing studies, they held off recommending SPIDER, though they saw promise in its precision and expected it to become more useful as the indexing of qualitative studies improves. **Sarah:** That fits with what we know about indexing. If qualitative studies often leave the word "qualitative" out of their titles and abstracts, a tight frame will lose some of them. **Kiffer:** Right. My own view is that SPIDER is very useful for thinking a question through, and the search strings themselves should cast a wider net, with validated filters, grey literature, and a check of reference lists. **Sarah:** Now the counting. The frequency effect size in a qualitative meta-summary is the proportion of included studies that report a given finding. Give me a case. **Kiffer:** Here's a made-up review of qualitative research on loneliness among international students. It includes twelve papers. One finding is that students describe loneliness as sharpest during holidays, when local classmates go home. Six of the twelve papers report it. One team was especially productive. Four of those six papers are theirs, all written from interviews with the same thirty students, and they have no other papers in the review. So here's the question. What's the frequency effect size for the holiday finding? Take a few seconds. *(Pause)* **Kiffer:** The quick answer is six divided by twelve, which is fifty percent. Sarah, does that number hold up? **Sarah:** It doesn't, because of that one team. Their four papers come from one study with one sample, so they're a single source of evidence counted four times. Sandelowski and Barroso, who developed meta-summary, count papers from a common parent study with overlapping samples once. So the four papers become one. The numerator drops from six to three, and the denominator drops from twelve to nine. Three divided by nine is one third, about thirty-three percent. **Kiffer:** That's right. The finding went from half the evidence to a third of it, once you count samples. **Sarah:** And the intensity effect size works at the level of each study. It's the proportion of all the distinct findings in the synthesis that one study contains. So if the synthesis ends with fifteen distinct findings and a rich ethnography contains nine of them, its intensity is nine divided by fifteen, sixty percent. **Kiffer:** Yes, and a brief report with three findings would score twenty percent, so intensity shows which studies carry the synthesis. What neither number gives you is confidence. That's a separate judgement, and it's the job of GRADE-CERQual, which looks at methodological limitations, coherence, adequacy of data, and relevance. Our productive team is an adequacy problem, because there's less independent data than the paper count suggests. **Sarah:** And possibly a relevance problem. If most of those studies interviewed students at large universities in big cities, the finding may say little about students at small colleges. **Kiffer:** Good, and there's one more check. If several of the original studies asked students directly about holidays, the finding recurs partly because of the interview guides. Frequency across studies can reflect what researchers asked as much as what students experienced. So report the frequency, and judge confidence separately. **Sarah:** Question four. How do you know whether your data can answer your question? I'd like to try this one myself. **Kiffer:** Here's a made-up project. A team has done twenty-two interviews with family caregivers of people living with dementia, one interview per person, all recruited through two caregiver support groups in one city. Their draft question is, "Does caregiver loneliness increase as dementia progresses?" Which of the four screens does that draft strain most, and why? Before you answer, Sarah, let's give everyone a few seconds to try it. *(Pause)* **Sarah:** Empirical answerability. "Increase" asks about amount and about change over time. One interview per person can't show anyone's loneliness rising, and twenty-two people found through support groups can't tell us how much loneliness caregivers have in general. So I'd make it comparative. I'd group the caregivers by stage of dementia, early, middle and late, then by whether they're a spouse or an adult child, then by whether they live with the person they care for, and compare how each group describes its loneliness. **Kiffer:** Before you commit to that design, count how many groups you've made. **Sarah:** Three stages times two relationships times two living arrangements. That's twelve groups. Twenty-two divided by twelve is fewer than two people per group, and some groups will end up with one person or none. That's the subgroup trap. I can't characterize a kind of experience from one person. **Kiffer:** So which part of your idea would you keep? **Sarah:** The interest in change. I'd drop the splits and ask about process. "How do family caregivers of people living with dementia describe changes in their loneliness over the course of caregiving, and what do they identify as turning points?" People can tell you about change in a single interview, even though I can't measure it. **Kiffer:** That's a process question, and the data can support it. Two caveats belong in the write-up. Accounts of change are told from where the person stands now, so they're reconstructions. And everyone came through a support group, so the study hears from caregivers who were connected enough to join one. That raises the question of who's missing, which is where Lesson three picks up. **Sarah:** And if someone does want to know how common caregiver loneliness is, that needs a survey with a probability sample. Statistics Canada's Canadian Social Survey, for example, asked about loneliness in August and September of 2021, and thirteen percent of people aged fifteen and older said they always or often felt lonely. **Kiffer:** That's a prevalence answer, and it took a national survey to produce it. The twenty-two interviews can do something that survey item can't, which is show what loneliness is like for these caregivers and how they think it came about. **Sarah:** Last screen, ethics. Suppose that halfway through, several caregivers start talking about grieving a person who's still alive, and the team wants to add questions about that grief. Can they just add them? **Kiffer:** In Canada, the Tri-Council Policy Statement, TCPS 2, has an article on emergent design, Article 10.5. Changes that leave the approved design essentially intact need no further review. Changes that could alter the level of risk to participants need research ethics board approval before they're used. **Sarah:** So rewording a confusing question is the first kind. Adding questions about grief could plausibly change the risk, so that goes back to the board before anyone asks them, along with a check that the distress protocol still fits the new questions. **Kiffer:** Yes, and because institutions handle amendments in slightly different ways, I'd contact your own ethics office early, well before the interviews where you hope to use the new questions. **Sarah:** Let's pull it together with three things to take away. First, exploratory work still carries theory. Name your sensitizing concepts, write down what you expect, and look hardest for the cases that don't fit. **Kiffer:** Second, a paradigm is a set of commitments that the rest of the methods section has to keep. Say what you treat the accounts as, what counts as a finding, and what role you played, and check that your coding, counting and claims match those answers. **Sarah:** And third, match every claim to the evidence underneath it. Papers from one sample count once, how often a finding recurs is separate from how much confidence it deserves, and twenty-two interviews can describe a process without telling you how common it is. **Kiffer:** If you'd like more practice, go back to the caregiver project and write a short methods paragraph for the rewritten question, with its paradigm, its design and the two caveats. Then appraise your own paragraph the way we appraised the invented one. **Sarah:** Next time, it's Lesson three, Sampling in Qualitative Research, where we look at who ends up in a study and what that means for the claims it can make. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson three.