AI Ed Wiki logoAI Ed Wiki
Use with AI

📄 Research Article

Student engagement with ChatGPT for educational tasks: Effects of inoculation training on verification intentions and behavior

● high · 2026-08-16 · ai-literacy, cognitive-offloading, trust-calibration, misinformation, higher-ed, language-learning, student-engagement

Synthesis: Vu, Cummings, and Park (2026) investigate whether an inoculation message can enhance students' intentions to verify ChatGPT-provided information and encourage actual verification behavior. Using a mixed factorial design (2×2×2: domestic vs. international Language Learning student status × inoculated vs. non-inoculated × pre-test vs. post-test, with a 48-hour gap between parts) with 100 U.S.-based students (40 domestic, 60 international EFL; mean age 20.86, SD = 2.75; 57% Asian), a generic forewarning inoculation message (relying on the motivational-threat mechanism rather than detailed refutations, to avoid reactance) was shown immediately before two tasks — an academic-source-summary task (find a scholarly article on social media and mental health) and a Math Education quiz exposing ChatGPT's known weaknesses in exponentiation and large-number multiplication. Screen-recorded verification behavior showed that inoculated students were significantly more likely to verify the academic-source-summary task (M = 0.34 vs. 0.18; F(1,96) = 4.85, p = .030, partial η² = .05), even though self-reported verification intentions (7-point scales) were not significantly affected by inoculation (F(1,88) = 0.12, p = .732), highlighting a gap between stated intention and enacted behavior.

Key Findings

Inoculation shifts behavior, not intention. A generic inoculation (forewarning) message increased students' actual verification of ChatGPT-generated academic-source summaries (inoculated M = 0.34 vs. control M = 0.18; F(1,96) = 4.85, p = .030, partial η² = .05) even though self-reported verification intentions were not significantly affected (main effect F(1,88) = 0.12, p = .732; time × inoculation F(1,88) = 0.02, p = .892) — a gap between stated intention and enacted behavior; intentions rose modestly for both conditions from pretest (M = 4.38 control / 4.23 inoculated) to posttest (M = 4.64 / 4.55).

International EFL students especially vulnerable. Students with high demand for language and academic assistance (international EFL students, here 60% of the sample) are particularly vulnerable to ChatGPT's risks, including hallucinations, misinformation, and limited advanced problem-solving; the study was designed to compare how this group verifies outputs against domestic students, who were defined as native/primary English speakers well-acquainted with U.S. academic traditions.

Task-dependent effects. Verification behavior was measured via screen-recording content analysis (binary use of external tools to cross-check answers, e.g., searching or pasting keywords into Google), and the effect of inoculation and student status depended on the type of task — a significant inoculation effect appeared for the academic-source-summary task (summarizing a scholarly article on social media and mental health) but not uniformly for the math quiz (exponentiation such as 3.2 to the power of 3.3, large-number multiplication, and a fraction word problem), underscoring the need for nuanced, context-aware approaches to promoting critical evaluation of AI output. A power analysis with G*Power 3.1 indicated the sample met requirements for the planned mixed ANOVAs and logistic regressions.

Moderating factors from qualitative data. Thematic analysis of open-ended responses surfaced awareness of hallucinations (n = 15), Trust in ChatGPT responses (n = 11), time constraints (n = 6), and training-modality suggestions (n = 7) as factors shaping inoculation effectiveness and verification behavior; perceived usefulness of the message was rated high (four items, M = 21.40, SD = 3.83, α = .80), and participants could enter a raffle for five $20 Amazon gift cards or receive SONA credits/$10 as incentives.

Pedagogical implication. The authors offer recommendations for designing brief, scalable inoculation messages and training tailored to different student groups to support more responsible and safe use of ChatGPT, aligning with the wiki's AI Literacy and Reducing AI Misuse concerns; limitations include a convenience sample of self-reported ChatGPT users (782 recruited, 100 valid cases), reliance on a single generic message rather than specific, detailed refutations, and one coder for the qualitative analysis.

Connected Concepts

Connected Articles

Citation

Vu, C. B., Cummings, J. J., & Park, D. Y. (2026). Student engagement with ChatGPT for educational tasks: Effects of inoculation training on verification intentions and behavior. Computers and Education Open, 100335. https://doi.org/10.1016/j.caeo.2026.100335