# Lesson 11: Analytic Induction, QCA and Decision Models *Companion-podcast transcript • Sarah & Kiffer* *Office Hours episode to listen to after working through the lesson* --- **Sarah:** Here's a claim to think about. A researcher reports that her explanation fits every one of her forty cases, with no exceptions at all. Should that make you more confident in her explanation, or less? **Kiffer:** It depends on what she did each time a case didn't fit. If she rewrote the explanation after every awkward case, or quietly narrowed what counted as a case, a perfect fit was always going to be the result. You'd want to see the trail of revisions before you believed it. **Sarah:** And I'd want to know who the forty are. If every one of them had the outcome, she's never checked whether her conditions turn up in people who didn't have it. **Kiffer:** That second worry goes back to a critique W. S. Robinson published in 1951, and the methods we're about to talk about are still answering it. **Sarah:** Welcome back to Office Hours. I'm Sarah. **Kiffer:** And I'm Kiffer. That puzzle about a perfect fit runs right through Lesson eleven, Analytic Induction, QCA and Decision Models, so let's start with it. **Sarah:** Question one. In analytic induction, when a case doesn't fit, you either revise the hypothesis or redefine the phenomenon. How do you tell a principled redefinition from dropping the cases you don't like? **Kiffer:** Let's work through an example, and every case in it is invented. Picture a weekly community lunch for older adults in a mid-sized town. The phenomenon is dropping out, defined as attending at least three times and then not returning within two months. The first hypothesis is that people drop out when they don't have a real conversation with anyone during their first three visits. **Sarah:** And the first case fits. A retired man who sat alone three times and stopped coming. **Kiffer:** Right. The second case is a woman who made a friend on her very first visit, someone she now phones every week. She stopped coming anyway, because the bus route that served her building was cancelled. **Sarah:** So here's a question for everyone listening. Do you revise the hypothesis to take her in, or redefine the phenomenon so she's no longer a case? Take a few seconds, and pause the audio if you'd like more time. *(Pause)* **Kiffer:** I'd redefine, and I'd write down why. Her leaving came from losing transport, which is a different process from losing interest. If you tried to revise, the hypothesis would become "no conversation, or a cancelled bus route," and once an "or" goes into the hypothesis, neither part on its own is necessary anymore. You've lost the one set of conditions, shared by every case, that the method aims at. **Sarah:** And that kind of "or" is what QCA calls equifinality, two separate paths to the same outcome. QCA is built to report several paths, and analytic induction is built to find one. So the redefined phenomenon here is something like disengagement, meaning dropping out while you can still get there. **Kiffer:** Yes, and there's a real precedent. Donald Cressey began with the legal category of embezzlement and moved to a category he called the criminal violation of trust. It took in similar frauds whatever the legal charge, which widened the category. It also required that the person had accepted the position in good faith, which narrowed it, because someone who took a job planning to steal was no longer a case. His approach followed an argument in criminology at the time, that researchers should group acts that are reasonably alike, however the law happens to name them. **Sarah:** That runs straight into a common objection to the method, that the revision step has no brakes. Anyone can claim their redefinition was principled. How would a reader check? **Kiffer:** Nobody can check it perfectly, so I'd use three habits. Write the redefinition down with a reason you could have given before you met the awkward case. List every case the new definition removes. And keep the removed cases, because they may start a second analysis, here one about how transport cuts end people's participation. Setting your revision rules before you start, as the lesson suggests, makes the first habit much easier to keep. **Sarah:** Now the objection Robinson is best known for. Every case so far is someone who dropped out. If we only ever talk to people who left, we can find what they all had in common, which speaks to necessity. We learn nothing about people who had the same experience and kept coming. **Kiffer:** So we go looking at people who stayed. The third invented case is a man who had no real conversation in his first three visits, and eight months later he's still coming every week. The hypothesis says he should have left. **Sarah:** So the condition is present and the outcome is absent. He shows that having no conversation isn't sufficient. **Kiffer:** Exactly. When you read his transcript, he says he started helping set up tables on his second visit, and the coordinator counts on him. So the revision adds a condition: people drop out when they have no conversation in their first three visits and have no role that ties them to the program. We'd never have found that by sampling only people who left. That's Robinson's sampling objection worked through on a single case. **Sarah:** One more thing bothers me about the no-exceptions rule. With interview data, some misfits are coding problems. A transcript is ambiguous, or I misread it. **Kiffer:** That's a fair worry, and it's why I'd re-read every negative case before revising anything. Ask whether it's a true misfit or a coding call you'd make differently on a second reading. Analytic induction treats every misfit as meaningful, so the coding has to be careful enough to deserve that treatment. **Sarah:** Question two. Reading and minimizing a truth table. I'd like to try this one myself, because I know where I tend to slip. **Kiffer:** Go ahead. Here's a made-up setup. Fourteen invented participants, and the outcome is whether they joined a neighbourhood walking group. Three conditions, each coded yes or no: retired, moved to the area in the past two years, and personally invited by someone they knew. **Sarah:** With three conditions, that's two to the power of three, so eight rows. Three rows show the outcome every time. Retired, new to the area and invited: three people, all three joined. Retired, long-time resident and invited: two people, both joined. Working, new to the area and invited: two people, both joined. Then the rest. Two rows are logical remainders with nobody in them: working, long-time resident and invited, and working, long-time resident and not invited. The other three not-invited rows hold seven people, and only one of them joined, a retired long-time resident who shares a row with people who didn't join. **Kiffer:** Before Sarah merges anything, try it yourself, using the rows she just read out. Which of the three outcome rows can be merged, and what's left? Take a few seconds. *(Pause)* **Sarah:** All three outcome rows have invited in common. The second and third rows both produce joining, so I merge them, drop retired and drop moved, and I'm left with invited. Invited is sufficient. **Kiffer:** Before you write that down, count how many conditions differ between those two rows. **Sarah:** The second row is retired and long-time. The third is working and new. So they differ on two conditions, and the rule only lets me merge rows that differ on exactly one. **Kiffer:** Right. Merging two rows that differ on two conditions quietly assumes the outcome for the other combinations of those two conditions. Hold onto that, because it comes back in a minute. Try again with the one-difference rule. **Sarah:** The first and second rows differ only on moving, so moving drops out and I get retired and invited. The first and third differ only on retirement, so I get new to the area and invited. The solution is retired and invited, or new to the area and invited. **Kiffer:** That's the conservative solution, and it uses only rows with people in them. What are the consistency and coverage for each path? **Sarah:** Retired and invited covers five people, and all five joined, so consistency is five divided by five, which is 1.0. Eight people joined in total, so coverage is five divided by eight, 0.625. The second path has the same numbers. Together the two paths cover seven of the eight people who joined. **Kiffer:** Now here's the twist. If you asked the software for the parsimonious solution, it would hand you the answer you first gave, invited on its own. **Sarah:** How? I got there by breaking the rule. **Kiffer:** Think back to what I said your shortcut was quietly assuming. Which rows does it need? **Sarah:** The other combinations of retired and moved, with invited. Retired, new and invited is already there with people in it. The missing one is working, long-time resident and invited, which is empty. So the software treats that empty row as if it would show the outcome, and once it's in, every merge follows the one-difference rule and the whole thing reduces to invited. **Kiffer:** That's it. The software makes the same assumption your shortcut made, and it makes it openly. So the parsimonious solution makes a claim about a kind of person nobody interviewed. **Sarah:** And whether that's reasonable is a question of theory. A working person who's lived in the neighbourhood for twenty years might have plenty of reasons to turn down an invitation. **Kiffer:** Which is the judgement the intermediate solution asks you to make. It allows that simplification only if theory gives you a reason to expect the empty row to show the outcome. Whatever you report, name the remainders you used. Then go back to the one joiner neither path explains, the retired long-time resident who joined without an invitation. That transcript is where the next round of reading starts. **Sarah:** Question three. Necessity and sufficiency. I can recite the definitions, and I still flip the direction when I'm looking at real data. The classic slip is testing necessity and dividing by the number of people who have the condition, which hands you the sufficiency number by mistake. **Kiffer:** A check that helps: for sufficiency, start from the cases that have the condition and look forward, asking whether they all show the outcome. For necessity, start from the cases with the outcome and look back, asking whether they all had the condition. So sufficiency consistency divides by the condition cases, and necessity consistency divides by the outcome cases. **Sarah:** Okay. Then give me a hard one. **Kiffer:** Twenty invented participants, and nine of them reconnected with an old friend in the past year. Condition one is owning a cell phone. Nineteen of the twenty do, including all nine who reconnected. Condition two is having an old friend living within an hour's drive. Ten of the twenty do, including eight of the nine who reconnected. Which condition passes the 0.90 threshold for necessity, and should you believe it? Take a few seconds. *(Pause)* **Sarah:** I'll start from the nine who reconnected. All nine own a cell phone, so nine divided by nine is 1.0, which passes. Eight of the nine have a nearby friend, and eight divided by nine is about 0.89, so it just misses. **Kiffer:** The arithmetic's right. Now look at the cell phone from the other side. Of the eleven people who didn't reconnect, how many own one? **Sarah:** Ten. Ten divided by eleven is about 0.91. So owning a cell phone also passes as a necessary condition for not reconnecting. That's absurd. **Kiffer:** It is, and QCA methodologists call this a trivial necessary condition. The phone is close to a constant, so almost any group of cases you pick falls inside it. Schneider and Wagemann describe a measure called relevance of necessity for exactly this problem, and here it would come out very low. **Sarah:** So a quick check, before I trust any necessary condition, is to count how many people lack it at all. For the phone, that's one person out of twenty. And the nearby friend, which is the condition that actually tells us something, misses the threshold by one person. **Kiffer:** Which means the decision turns on one transcript. With nine outcome cases, each case moves the consistency by about 0.11. So read the one person who reconnected without a nearby friend. Suppose she reconnected by video with a friend across the country. Then the useful condition may be having an old friend you can still reach, and that's a coding rule you'd have to apply to all twenty cases. **Sarah:** Though that's the same danger as question one. I could keep adjusting the condition until it clears 0.90. **Kiffer:** You could, so I'd report both versions, the original condition at 0.89 and the revised one with its new number, and say plainly that the revision came after you'd seen the data. Then ask the triviality question again, because almost everyone may have an old friend they can still reach. The same question is worth asking of any condition that passes necessity in a twenty-case table, including the caregiving result in the lesson. **Sarah:** Question four. Decision trees and the eighty percent rule. Out-of-sample accuracy sounds like a clean test. Where does it go wrong? **Kiffer:** Here's an invented case. A researcher uses six transcripts to build a tree for the choice between reaching out and withdrawing, and tests it on ten held-out transcripts. The tree classifies eight of the ten correctly. That's eighty percent, right at the threshold. Before you accept it, what one other number would you want to see? Take a few seconds. *(Pause)* **Sarah:** How many of the ten actually withdrew. If most of them did, a tree with no questions at all, one that says everyone withdraws, would also score well. **Kiffer:** And in this case eight of the ten withdrew. So the rule that everyone withdraws also gets eight of ten, the same eighty percent, with no decision logic at all. Broken down, the tree got seven of the eight people who withdrew and one of the two who reached out. **Sarah:** So it's no better than the base rate, and it's weakest on reaching out, which is the choice an intervention would be trying to encourage. **Kiffer:** That's the reading I'd give it. Report accuracy for each outcome separately, next to the accuracy of always guessing the most common outcome. And with ten test cases, each transcript is worth ten percentage points, so passing or failing comes down to a single person. **Sarah:** And then the natural move is to revise the tree using the two cases it got wrong. **Kiffer:** That's fine, as long as you're clear about what the new number means. Say the researcher adds a branch that fixes those two cases and re-tests on the same ten. The tree now scores ten of ten. That one hundred percent is an in-sample number, because those ten cases helped build the tree. **Sarah:** So with twenty transcripts, I'd set some aside at the very start. Build on six, test on seven, and keep seven untouched for a final check after any revision. And I'd report the first test result alongside the final one. **Kiffer:** That's the split I'd use if the data allow it. Another option, closer to how the method was designed, is to test the revised tree on a fresh set of participants who played no part in building it. **Sarah:** Okay, here's where Kiffer and I don't fully agree. The idea from the lesson is that building a truth table or a tree disciplines interpretation and makes it auditable. I think it can also hide interpretation. When I code a transcript as existential loneliness, one or zero, the hardest judgement in the whole analysis disappears into a single cell. The table looks objective, and the reading behind it is out of sight. **Kiffer:** I see it the other way, at least partly. A theme summary keeps its judgements in prose, where they're hard to point at. A truth table puts each judgement in a cell that another reader can challenge. If you and I disagree about one participant, we can find the exact cell and argue about the passage behind it. **Sarah:** Only if readers go back to the transcripts, and the evidence suggests many studies don't make that easy. A 2021 systematic review of QCA studies of public health interventions looked at twenty-six studies. The reviewers judged that twelve of them showed too little familiarity with their cases, and only five met all six of their quality criteria, which included reporting the truth table and the raw data. **Kiffer:** That's a real problem, and I'd read it as a problem of practice. The review treats knowing your cases as part of doing QCA properly, and it faults the studies that fell short on it. What I'd defend is the formal object kept beside the cases, with its coding rules published. **Sarah:** Then there's the language. Saying a combination is sufficient for existential loneliness sounds causal to an epidemiology reader, even when the row holds three synthetic transcripts. **Kiffer:** I agree with you on the wording. I'd write that in these twenty cases, everyone with this combination described the outcome, and I'd give the number of cases behind each path every time I report a consistency score. **Sarah:** So where do we land? **Kiffer:** Here's a rule I think we can share. Every one and zero in the table should trace back to a written coding rule and a quoted passage. Flag the borderline codings, rerun the analysis with those cells flipped, and report whether the solution holds. If it changes, report that too, because it tells the reader how much the result depends on a few judgement calls. **Sarah:** I can live with that. I'm still warier of the zeros and ones than Kiffer is, but a table that's been tested that way is one I'd trust. **Kiffer:** Let's pull it together with three things to take away from this episode. **Sarah:** First, in analytic induction, write down every revision and every redefinition with its reason, and go looking for cases where the outcome is absent. Without them, you can speak to necessity and say very little about sufficiency. **Kiffer:** Second, a QCA solution is only as good as what sits behind it. Know which remainders it used, check whether a necessary condition is close to a constant, and remember that in a small table one case can move a result across a threshold. **Sarah:** And third, judge a decision tree's accuracy next to the base rate, report accuracy for each outcome, and keep the cases you test on separate from the cases you revise with. **Kiffer:** If you'd like more practice, go back to the walking group truth table from this episode. Put two people in the empty row of working, long-time residents who were invited, and work out the conservative and parsimonious solutions twice, once with both of them joining and once with neither joining. **Sarah:** Next time, it's Lesson twelve, Computational Text Analysis, Cultural Domain Analysis and LLM-Assisted Coding, where we ask how far software and language models can carry this kind of careful reading. **Kiffer:** Take care, everyone. **Sarah:** See you in Lesson twelve.