Student engagement with ChatGPT for educational tasks: Effects of inoculation training on verification intentions and behavior

Synthesis: Vu, Cummings, and Park (2026) investigate whether an inoculation message can enhance students' intentions to verify ChatGPT-provided information and encourage actual verification behavior. Using a mixed factorial design (2×2×2: domestic vs. international Language Learning student status × inoculated vs. non-inoculated × pre-test vs. post-test, with a 48-hour gap between parts) with 100 U.S.-based students (40 domestic, 60 international EFL; mean age 20.86, SD = 2.75; 57% Asian), a generic forewarning inoculation message (relying on the motivational-threat mechanism rather than detailed refutations, to avoid reactance) was shown immediately before two tasks — an academic-source-summary task (find a scholarly article on social media and mental health) and a Math Education quiz exposing ChatGPT's known weaknesses in exponentiation and large-number multiplication. Screen-recorded verification behavior showed that inoculated students were significantly more likely to verify the academic-source-summary task (M = 0.34 vs. 0.18; F(1,96) = 4.85, p = .030, partial η² = .05), even though self-reported verification intentions (7-point scales) were not significantly affected by inoculation (F(1,88) = 0.12, p = .732), highlighting a gap between stated intention and enacted behavior.

Key Findings

Inoculation shifts behavior, not intention. A generic inoculation (forewarning) message increased students' actual verification of ChatGPT-generated academic-source summaries (inoculated M = 0.34 vs. control M = 0.18; F(1,96) = 4.85, p = .030, partial η² = .05) even though self-reported verification intentions were not significantly affected (main effect F(1,88) = 0.12, p = .732; time × inoculation F(1,88) = 0.02, p = .892) — a gap between stated intention and enacted behavior; intentions rose modestly for both conditions from pretest (M = 4.38 control / 4.23 inoculated) to posttest (M = 4.64 / 4.55).

International EFL students especially vulnerable. Students with high demand for language and academic assistance (international EFL students, here 60% of the sample) are particularly vulnerable to ChatGPT's risks, including hallucinations, misinformation, and limited advanced Problem Solving; the study was designed to compare how this group verifies outputs against domestic students, who were defined as native/primary English speakers well-acquainted with U.S. academic traditions.

Task-dependent effects. Verification behavior was measured via screen-recording content analysis (binary use of external tools to cross-check answers, e.g., searching or pasting keywords into Google), and the effect of inoculation and student status depended on the type of task — a significant inoculation effect appeared for the academic-source-summary task (summarizing a scholarly article on social media and mental health) but not uniformly for the math quiz (exponentiation such as 3.2 to the power of 3.3, large-number multiplication, and a fraction word problem), underscoring the need for nuanced, context-aware approaches to promoting critical evaluation of AI output. A power analysis with G*Power 3.1 indicated the sample met requirements for the planned mixed ANOVAs and logistic regressions.

Moderating factors from qualitative data. Thematic analysis of open-ended responses surfaced awareness of hallucinations (n = 15), Trust in ChatGPT responses (n = 11), time constraints (n = 6), and training-modality suggestions (n = 7) as factors shaping inoculation effectiveness and verification behavior; perceived usefulness of the message was rated high (four items, M = 21.40, SD = 3.83, α = .80), and participants could enter a raffle for five $20 Amazon gift cards or receive SONA credits/$10 as incentives.

Pedagogical implication. The authors offer recommendations for designing brief, scalable inoculation messages and training tailored to different student groups to support more responsible and safe use of ChatGPT, aligning with the knowledge base's AI Literacy and Reducing AI Misuse concerns; limitations include a convenience sample of self-reported ChatGPT users (782 recruited, 100 valid cases), reliance on a single generic message rather than specific, detailed refutations, and one coder for the qualitative analysis.

What this means for practice

• Instructors. Deliver a short forewarning message before any task that depends on ChatGPT: inoculation nearly doubled verification of a scholarly-source task (M = 0.34 vs. 0.18, p = .030) for a few minutes of class time.
• Instructors. Do not treat students' stated intentions as evidence that training worked — self-reported verification intentions were unchanged (F(1, 88) = 0.12, p = .732) — and assess what students actually do instead.
• Learners. Cross-check AI answers with an external tool on tasks where the model is known to fail, such as locating a real scholarly article, rather than accepting a plausible summary.
• Faculty developers. Embed inoculation and verification training in support for international EFL students, who made up 60% of the sample and have the highest demand for language and academic assistance.

Limitations

• Of 782 students recruited, only 100 valid cases were analyzed (9 pilot plus 91 main-study cases), a convenience sample restricted to U.S. students who already self-reported using ChatGPT.
• Participants completed six-minute tasks in a screen-recorded session with raffle, SONA, or cash incentives, conditions far from unsupervised homework.
• The intervention was one generic forewarning message rather than detailed refutations, and its behavioral effect was task-specific: significant for the academic-source-summary task but not uniform on the math quiz.
• Verification behavior was coded from screen recordings with a single coder, and intentions and message usefulness were self-reported (usefulness M = 21.40, α = .80).

Citation

Vu, C. B., Cummings, J. J., & Park, D. Y. (2026). Student engagement with ChatGPT for educational tasks: Effects of inoculation training on verification intentions and behavior. Computers and Education Open, 100335. https://doi.org/10.1016/j.caeo.2026.100335