# Lesson 7: AI as a Thinking Partner: Generating and Testing Ideas

*Companion-podcast transcript • Sarah & Kiffer*  
*Office Hours episode to listen to after working through the lesson*

---

**Sarah:** Here's a made-up example. A public health student asks an assistant for thirty ways to get more older adults to use cooling centres during a heat warning.

**Kiffer:** The list comes back quickly and looks impressive. Then the student merges the near-duplicates, and there are nine distinct ideas.

**Sarah:** Radio, posters, text alerts and social media are four versions of telling people the centre exists.

**Kiffer:** And a free shuttle shows up three times. So the student asked for thirty ideas and has nine to choose from.

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This is Lesson seven, AI as a Thinking Partner: Generating and Testing Ideas. We're starting with the research on an assistant's ideas, because the evidence can look contradictory.

**Sarah:** It does. In the 2023 product-ideas study, a language model produced two hundred ideas in about fifteen minutes, and potential consumers rated its ideas higher on average than the business students' ideas. In the study with more than one hundred researchers, reviewers rated the model's research ideas as more novel than the experts' ideas. Then the 2025 brainstorming experiments found that sets of ideas produced with ChatGPT's help were significantly less diverse in thirty-seven of forty-five comparisons. So which is it?

**Kiffer:** It's both, because the studies measure different things. The first two studies judge ideas one at a time. The brainstorming experiments judge the whole set. A model can write many individually appealing ideas that all sit in the same neighbourhood. In the toy task, nine participants working with the assistant independently gave their toy the same name.

**Sarah:** And then there's the follow-up, where experts actually carried out the research ideas.

**Kiffer:** Right. Forty-three experts each spent more than one hundred hours executing an idea, and once the projects were finished the model's advantage disappeared. I'd take two working rules from all this. Ask for many ideas, because they're cheap and some are very good. And treat the whole set with some suspicion, because it clusters, and an idea that reads well is still a candidate until it has been tested.

**Sarah:** The split between producing many answers and narrowing them down goes back a long way, doesn't it?

**Kiffer:** It does. The psychologist J. P. Guilford, whose 1950 address on creativity helped establish its scientific study, separated divergent thinking from convergent thinking. What's new is that the divergent half has become very cheap. Let's make the clustering concrete with a made-up example that starts from a real event. After the extreme heat in British Columbia from June twenty-fifth to July first, 2021, six hundred and nineteen deaths were identified as heat-related. A death review panel of the B.C. Coroners Service reported in 2022 that sixty-seven percent of those who died were aged seventy or older, more than half lived alone, and in ninety-eight percent of the deaths the heat injury happened indoors, in a residence.

**Sarah:** So here's the made-up part. A public health student in a small B.C. city asks an assistant for thirty ways to get more older adults to use cooling centres during a heat warning. The list comes back quickly and looks impressive. When the student merges the near-duplicates, there are nine distinct ideas. Radio, posters, text alerts and social media are four versions of telling people the centre exists, and a free shuttle shows up three times.

**Kiffer:** That's the clustering you'd expect. Before we fix the prompt, here's a question for everyone. Given what the review found about who died and where, what's limiting about the goal the student gave the assistant? Take a few seconds.

*(Pause)*

**Sarah:** The goal assumes people will leave home and go somewhere. The people at greatest risk were mostly older, often living alone, and at home. And some of them may not have been able to reach a centre at all.

**Kiffer:** That's it. The review itself noted that people with mobility issues may not be able to reach cooling centres, and said that solutions bringing cooling to people who can't easily leave home will be required. It also pointed to heat-informed checks on older adults and people who live alone. The assistant answered the question it was given. So the first place to widen the range of ideas is the goal itself, before any of the techniques for spreading out the list.

**Sarah:** So a second request might be, list twenty ways to keep older adults who live alone safe during extreme heat, five each from a building manager, a home support worker, a neighbour on the same floor and a paramedic, and don't repeat any of these nine ideas.

**Kiffer:** And in our made-up version, fourteen of those twenty are new. They include a floor-by-floor check by the building manager, a cool common room in the building, home support workers checking the indoor temperature at each visit, and neighbours pairing up to check on each other twice a day.

**Sarah:** Nine distinct ideas out of thirty is thirty percent. Fourteen new ones out of twenty is seventy percent. So the better-aimed request did more than twice as well.

**Kiffer:** And a third request, the wider goal again in a fresh conversation, adds only two new ideas out of thirty, about seven percent. That's the signal to stop generating and start choosing, with criteria the student sets before looking at any ratings.

**Sarah:** Question two. If an assistant criticizes on request and backs down when you push, what is its criticism worth? I'll admit this one bothers me. It can feel as if the assistant will say whatever I've set it up to say.

**Kiffer:** That worry has evidence behind it. Sharma and colleagues tested five assistants in 2023 and found sycophancy in all five. Feedback got warmer when the user said they had written the work, and assistants often abandoned a correct answer when the user asked, are you sure? And in April 2025, OpenAI withdrew an update to ChatGPT within a few days, after users reported it had become excessively flattering. So let's run a made-up example and see what survives.

**Sarah:** Okay. A student wants to learn why older adults in a rural district don't use a free shuttle to the regional hospital. The plan is an online survey promoted on the district's social media page. First the student asks, I've designed this and I think it's solid, what do you think? The assistant praises the plan and suggests shorter questions.

**Kiffer:** Then, in a fresh conversation, with no mention of who wrote it, the student asks for the five strongest objections, ranked, each with the evidence that would answer it. In summary, here's what comes back. One, the people the question is about, older adults who don't use the shuttle, may be the least likely to follow that page or answer an online survey. Two, the response may be too small to say much. Three, the questions may assume that people know the shuttle exists. Four, the wording may lead people. Five, more research is needed on rural transportation.

**Sarah:** And then the student pushes back on the first objection. Are you sure? Most seniors are online now. The assistant apologizes and says an online survey should reach most older adults.

**Kiffer:** So here's the question for everyone. After that reversal, how much should the student's view of the first objection change? Take a few seconds.

*(Pause)*

**Kiffer:** It shouldn't change at all. The reversal came with no new reason, so it carries no information. The mechanism in the first objection is the same as it was a minute earlier: the survey reaches people through a channel that non-users may lack. Whether that matters in this district depends on a local fact the assistant can't know, which is how many older adults there use the internet and follow that page.

**Sarah:** So the assistant's verdict changed with the way the student asked, and the reason behind the objection stayed the same.

**Kiffer:** That's how I'd put it. I'd judge each objection by its reason and give the assistant's verdict no weight on its own. A useful objection names a mechanism, says which part of the conclusion it threatens and says what evidence would settle it. The first objection does all three. The fifth, more research is needed, does none of them.

**Sarah:** There's still a problem, though. I asked for five and got five. If the plan had one real weakness, I'd still get four more.

**Kiffer:** True, and that's worth building into the prompt. Ask the assistant how many of its objections it considers serious, and tell it that fewer than five is an acceptable answer. When you disagree, ask for its strongest reason and for the information that would show the objection doesn't apply. Here, the student's best move is to keep the online survey and add an offline route, such as paper forms at the post office and the pharmacy counter, so that non-users end up in the sample.

**Sarah:** And the risk sits on my side as well as the assistant's. Lee and colleagues surveyed three hundred and nineteen knowledge workers, and the ones with more confidence in AI for a task reported applying less critical thinking to it.

**Kiffer:** Yes, with the caution that it's a survey at one point in time, so it can't show that confidence causes less thinking. People may simply rely on AI most for tasks they see as routine. It's still a good reason to write down your own view before you ask.

**Sarah:** Question three. When does a comparison group fail to deal with regression to the mean? This is the harder example, and I'd like to try it myself.

**Kiffer:** Go ahead. It builds on the fictional Cedar Valley Health Authority. Its Connector program refers older adults who score six or higher on the three-item UCLA Loneliness Scale, which runs from three to nine. Among participants who completed both measurements, the mean fell from 7.2 at referral to 6.0 at twelve weeks, and the obvious worry was regression to the mean, because people were picked for high scores.

**Sarah:** And the team planned to get a comparison from the second-wave clinics.

**Kiffer:** Right, so here's a made-up extension. A draft summary adds a comparison from the second-wave clinics, during the year before the program starts there. Among all adults sixty-five and older who answered the clinics' health survey twice, twelve weeks apart, the mean went from 5.1 to 5.0. The draft says the program lowered loneliness by about 1.1 points and that the comparison group rules out regression to the mean. It's pasted into an assistant with the question, can you confirm this reasoning is sound? The assistant says yes, and repeats the estimate of 1.1 points.

**Sarah:** So the question is whether that comparison really rules out regression to the mean.

**Kiffer:** Let's give everyone a chance to try it before you answer. Does this comparison group rule out regression to the mean? Take a few seconds.

*(Pause)*

**Sarah:** Okay. The participants fell 1.2 points, from 7.2 to 6.0. The comparison group fell 0.1, from 5.1 to 5.0. The difference is 1.1. The comparison group shows what happens over twelve weeks without the program, so it takes care of anything that would have happened anyway, and that includes regression to the mean. So I'd accept the assistant's answer, and maybe ask whether 1.1 points could be chance.

**Kiffer:** Let me ask one thing first. Why were the participants' first scores high?

**Sarah:** Because they were selected for being high. They had to score six or more to be referred. And the comparison group is everyone who answered a survey, whatever their score, with a mean around five. Nobody picked them on a bad day, so as a group they have nothing to regress from. I made the same mistake the assistant did. Having a comparison group isn't enough by itself.

**Kiffer:** Exactly. Regression to the mean comes from selecting people because of an extreme score, part of which reflects temporary circumstances and measurement error. A comparison group can account for it only when its members are selected the same way, from a similar population, and measured on the same schedule. Here, that would be people in the second-wave clinics who screened six or higher and then went twelve weeks without the program.

**Sarah:** So suppose we had that group. What are its numbers?

**Kiffer:** These are still made up. Say that group went from 7.1 to 6.5.

**Sarah:** Then they fell 0.6 points with no program at all. The participants fell 1.2. So a better estimate of the program's effect is 1.2 minus 0.6, which is 0.6 points. That's a little over half of the 1.1 in the draft.

**Kiffer:** And what's still unresolved?

**Sarah:** The scores only include people who completed both measurements, so it matters who dropped out. If connectors give the twelve-week measure, people may report less loneliness to someone who helped them. And season matters, so both groups have to be measured at the same time of year.

**Kiffer:** Good. Now notice why the assistant agreed. The question supplied the answer and asked for confirmation, which invites answer sycophancy, one of the four forms Sharma and colleagues described. A neutral version would say, list the conditions under which a comparison group accounts for regression to the mean, and say whether this comparison meets each one.

**Sarah:** Question four. When an assistant suggests a variable for a causal diagram, how do you tell what role it plays? The definitions seem clear on paper, and I still get them wrong.

**Kiffer:** Many people do. A good first clue to a variable's role is its timing. Here's a made-up question: among adults sixty-five and older, is taking up a regular volunteer role associated with loneliness one year later? An assistant is asked to list missing variables and classify each one. It suggests the number of new friendships a person makes during that year, and calls it a confounder because it's associated with both volunteering and loneliness.

**Sarah:** So here's the question for everyone. Are new friendships made during the year a confounder, a mediator or a collider? Take a few seconds.

*(Pause)*

**Kiffer:** They're a mediator. The friendships come after the person starts volunteering, volunteering plausibly causes some of them, and they plausibly lower loneliness. So they sit on the pathway from the exposure to the outcome, and adjusting for them would remove part of the effect the question asks about.

**Sarah:** And being associated with both is what a mediator looks like too.

**Kiffer:** Right. A confounder, a mediator and a collider can all be associated with both variables. Now change the timing. The number of close friends a person had before they started volunteering is a plausible confounder, because people with more friends may be more likely to be asked to volunteer, and those friendships may also keep them less lonely a year later. The concept is the same, and the timing gives it a different role.

**Sarah:** So the fix in the prompt is to make the assistant say when each variable happens.

**Kiffer:** Yes. Ask it to place every suggested variable in time, before the exposure, between the exposure and the outcome, or after the outcome, and to justify each role from that timing and the direction of its arrows. Then check every classification against the definitions yourself, because the assistant may still get some of them wrong.

**Sarah:** Okay. Now the part where we see things differently, which is simulated perspectives.

**Kiffer:** This means asking an assistant which concerns a group of interest holders, such as clinic staff or older adults, is likely to raise before you meet them. Sarah and I agree on the limits. Simulated views are never reported as anyone's actual views, and a rights holder's position, such as a First Nation's, is never simulated. Within those limits, I think it's useful preparation.

**Sarah:** My concern is with the preparation itself. Anchoring is the tendency for the first options you see to shape and limit what you consider afterward. If a team walks into a meeting having read a list of predicted concerns, I think they'll listen for those concerns and hear them. In the Cedar Valley case, the steering committee's main request, to include people under sixty-five who care for a spouse at home, didn't appear in the simulation at all.

**Kiffer:** But the team did hear it. I'd read that example the other way. Questions like how much time screening adds, and whether referrals lead anywhere, are predictable. Having an information sheet ready answers them quickly, which leaves more of the meeting for the things only those people can raise. I see good preparation as a way of respecting their time.

**Sarah:** That's fair for clinic staff. I'm more worried about older adults on a steering committee. A model's picture of what older adults think comes from general patterns in its training text. It can reproduce stereotypes, and it treats a varied group as if it held one view. And when schedules get tight, I think the simulation starts to feel like a substitute for the meeting.

**Kiffer:** That last risk is real, and I'd treat it as a firm line. If a simulated list is ever treated as people's input in place of a meeting, that's a form of tokenism. Where I'd disagree is on dropping the tool. I think a team that prepares well is more likely to be trusted, and marking every predicted concern as inferred and unconfirmed reduces the stereotype risk.

**Sarah:** Marking it reduces the risk that someone reports it as real. I'm less sure it stops the list shaping what the team hears. Could we agree on a habit for the meeting itself?

**Kiffer:** Let's try. Open every engagement with open questions, before anything prepared from a simulation goes on the table.

**Sarah:** And afterward, compare what people raised with what the simulation predicted, and record what it missed in the AI use log. Over time, that tells the team how much weight the simulation deserves.

**Kiffer:** I'd sign up for that. Use simulated concerns to prepare, start each meeting with open questions, and keep a record of what the simulation missed.

**Sarah:** I'm still warier of the tool than Kiffer is, and that's a rule I'd follow too.

**Kiffer:** Let's pull it together with three things to take away.

**Sarah:** First, an assistant's ideas can be strong one at a time and still cluster as a set. Widen the goal as well as the list, and stop generating when the new ideas run out.

**Kiffer:** Second, an assistant's verdicts move with the way you ask. Judge each objection by its mechanism and the evidence that would settle it, and treat a reversal under pushback that comes with no new reason as no information.

**Sarah:** And third, apply the definitions yourself. A comparison group accounts for regression to the mean only when it's selected the same way, and a variable's role in a causal diagram depends on when it happens and which way its arrows point.

**Kiffer:** If you'd like more practice, rework the comparison-group problem from this episode with your own made-up numbers. Change how the comparison group is selected, and watch what happens to the estimate of the program's effect.

**Sarah:** Next time, it's Lesson eight, AI for Data Analysis and Coding in R, where we look at describing data to an assistant, building an analysis in small steps and catching code that runs without error and still gives the wrong answer.

**Kiffer:** Take care, everyone.

**Sarah:** See you in Lesson eight.