From answer machine to collaborative partner: impact of structured generative AI training on scientific problem solving in STEM education

Synthesis: Huang and colleagues (2026) ran a randomized controlled trial with 95 undergraduate STEM students in China to test whether a 45-minute scaffolded training session changes how students use generative AI on difficult physics problems. Both groups solved the Conceptual Survey of Electricity and Magnetism unaided, then discussed their uncertain items with the chatbot Doubao and revised. The trained group revised to correct answers far more often (69.3% of their incorrect answers became correct, against 41.3% in the control group) and reached a higher posttest score with a large effect (d = 0.814), without reporting more total cognitive load — although the authors argue the trained group's marginally higher intrinsic load reflects productive struggle rather than overload.

Key Findings

1. Structured training improved AI-assisted revision, with a large effect. The trained group's posttest CSEM score was 83.29% (SD = 10.19) against 74.376% (SD = 11.723) for controls, t(93) = 3.964, p < 0.001, d = 0.814 — after the two groups had been shown equivalent on prior knowledge, SRL skills, and GenAI experience.
2. The gap was in revision quality more than revision volume. Trained students made fewer total revisions (254, M = 5.080) than controls (368, M = 8.178), yet converted 69.3% of their incorrect answers to correct ones where controls managed 41.3%; 40.0% of control-group revisions turned a correct answer into an incorrect one, against 16.5% for the trained group.
3. Total cognitive load did not differ between groups. Reported load was 3.925 (SD = 0.979) for the trained group and 3.767 (SD = 0.890) for controls (p = 0.414, d = 0.168), with no significant differences in germane or extraneous load.
4. Intrinsic load rose marginally in the trained group. It measured 4.700 (SD = 1.305) against 4.211 (SD = 1.375) in controls (p = 0.079, d = 0.365), which the authors read as strategic engagement with the problem rather than instructional overload.
5. Training attenuated the advantage of prior GenAI experience. The Group × GenAI experience interaction was the only significant moderator (β = −3.445, 95% CI [−7.137, −0.264], p = 0.034); the slope between GenAI experience and revision performance was 3.460 in the control group and 0.015 in the trained group.
6. Self-regulation and prior knowledge still predicted performance in both conditions. In the robust main-effects model (Magee's pseudo R² = 0.580), SRL skills (β = 2.556) and pretest CSEM score (β = 7.513) predicted revision performance, while GenAI experience was only marginal (β = 1.349, p = 0.100).
7. Trained students used more higher-order interaction strategies. Obtaining insight into problems rose from 6.6% to 21.1% of strategy instances (20.0% to 42.0% of students), generating new questions went from 0% to 14.0%, and image-only prompting — still dominant in both groups — fell slightly from 63.4% to 58.6%.

Training design: 45 minutes, three modules, two problem domains

The intervention was a single 45-minute session built on Zimmerman's three-phase self-regulated learning model and cognitive load theory: 10 minutes introducing what GenAI is and how it differs from search engines and homework-answer apps, 15 minutes demonstrating phase-specific prompting strategies and remediation tactics for hallucination, reasoning errors, and visual misreading, and 20 minutes of hands-on practice on two mechanics problems. The control group received 45 minutes of advanced Problem Solving strategy instruction with the same structure, the same problems, and no AI, delivered by a second instructor after a synchronization meeting.

Two design choices carry most of the study's internal validity. Students practiced on mechanics problems and were assessed on electromagnetism items, so the assessment probed transfer of strategy rather than familiarity with the practice domain. And both groups' interaction with Doubao was restricted to basic chat and image upload, with system prompts, web search, and plugins disabled — closing the gap between conditions that would otherwise come from unequal access to features. The design is deliberately short: the authors note that multi-week interventions of the kind used in comparable studies cannot hold learner characteristics constant while also manipulating instruction.

What the group differences did and did not cost

The revision-pattern data are the clearest evidence that the training changed behavior rather than effort. Controls revised more and revised worse, with 40.0% of their changes moving a correct answer toward an incorrect one — the pattern the authors describe as reactive capitulation to plausible-looking output. Trained students were also not merely more confident: the marginal rise in intrinsic load alongside no rise in extraneous load is offered as productive struggle, and the authors propose splitting intrinsic load into a task-imposed component and a productively engaged component that reflects deliberate strategy use.

That reading goes beyond what a single-session design can establish, and the paper says so: perceived-limitation detection was not measured, the load instrument is self-reported, and the constructive proposal is offered as a refinement for future work rather than a demonstrated mechanism. The alternative explanations the authors raise for the strategy shift — that students revised only when confident, or that the training simply handed them a script to follow — are left standing, with a transfer task on novel items named as the way to separate them.

Learner characteristics, moderation, and a construct proposed rather than tested

The robust models place learner characteristics in a specific relation to the intervention. SRL skills and prior knowledge predicted better revision performance regardless of group, which the authors read as metacognitive capacity still mattering in AI-mediated work. Prior GenAI experience predicted performance only in the control group: once students were taught how to interrogate the model, experience with the model stopped being the differentiator, which is the study's clearest result for equity: the returns to prior tool familiarity can be levelled by instruction.

To explain the qualitative patterns the authors propose complementarity recognition — awareness of the asymmetric capabilities of humans and AI, and of when to rely on which — while stating plainly that it was not measured and needs an instrument before it can be tested. The construct is worth noting because it names what the transcripts show: students who caught the model's visual errors and demanded a re-analysis were doing something the control group's highest-frequency strategy, asking the model to check answers, does not capture.

From snap-and-solve to strategic prompting

The transcripts are the study's most concrete contribution for teachers. The control group's dominant strategy was asking whether answers were correct; one student went further and instructed the model to state the correct answer to each question directly after grading, and the model complied, after which a visually misread graph produced a wrong answer the student did not catch. Trained students in the same situation named the error: one pointed out that a coil the model had called an open circuit was closed, prompting a corrected answer, and another corrected a graph-matching error by describing the required pattern of the voltmeter reading.

Both groups still led with image-only prompting — 82.2% of controls and 88.0% of trained students used it at least once — which the authors attribute to the local "snap-and-solve" culture of homework-answer apps rather than to the models themselves. That persistence is the practical warning in the paper: one training session diversified strategies at the top of the distribution without dislodging the habit underneath it.

What this means for practice

• Instructors. Teach the interaction, not the tool: a single session on what the model gets wrong and how to interrogate it changed revision quality more than access to the model did.
• Instructors. Require students to attempt problems before consulting GenAI, then have them and the model disagree explicitly — asking students to identify where the model's reasoning fails is what turned correct answers into corrections.
• Instructional designers. Build the trained behaviors into the task: phase-specific prompts for planning, problem insight, and self-checking are concrete, teachable moves rather than general "AI literacy" aims.
• Instructors. Expect prior tool experience to stop predicting performance once scaffolding is in place, so grouping or streaming by GenAI familiarity is not a valid proxy for who needs support.
• Researchers. Treat these effects as short-term. Same-day pretest and posttest on identical items, plus the control group's lack of AI guidance, leave the durability and the active ingredient both unresolved.

Limitations

• One session, one day, identical items. The pretest and posttest used the same CSEM items on the same day, so posttest gains may partly reflect memory of the uncertain items and the motivation to consult the model about them; the design cannot support claims about durable learning or transfer.
• The control condition is not an active AI control. Controls received no AI guidance, so the effect may come from the absence of guidance rather than from the training's specific components, which were not tested separately.
• The strategy shifts have two live explanations. Scripted behavior from the training and a selection effect (revising only when confident) were not distinguished from genuine changes in calibrating trust in trust.
• 95 students, one university, one culture. Sophomore STEM students at a top-tier university in central China, in a single-session design with self-reported load; the authors call for replication with larger samples, since the intrinsic-load and interaction findings are preliminary.
• The tool is named and the version is not. All interactions ran on Doubao, a Chinese model chosen for local familiarity, and the paper states no model version and no data-collection window — only the 2026 submission and acceptance dates — so how far the findings carry to other models or later versions is untested.
• Both cognitive constructs are measured by questionnaire or inferred from transcripts. Cognitive load is self-reported, and complementarity recognition, proposed as the mechanism, was never measured.

Citation

Huang, B., Xie, L., Liu, Y., Liu, X., Tang, H., Feng, X., Qiao, C., Guo, Q., Wu, C., & Bao, L. (2026). From answer machine to collaborative partner: impact of structured generative AI training on scientific problem solving in STEM education. International Journal of STEM Education, 13(62).