When Teachers Use AI Chatbots and Are Trained for It: Impact on Learning Design Quality and Cognitive Effort

Synthesis: In a within-subjects study, 13 higher-education teachers designed learning activities across three conditions — no AI, with a chatbot, and with a chatbot after receiving training on pedagogically grounded AI-interaction strategies. AI access significantly improved design quality (most clearly by raising attainment of higher-order Bloom's tasks) and lowered perceived cognitive effort; the training did not further improve design quality but slightly increased cognitive effort, likely reflecting deliberate application of newly learned strategies. Teachers rated the training as useful and reported strong adoption intentions.

Relevance to AI in Education: This study empirically examines whether and how AI support and guidance shift teachers' Learning Design quality and the cognitive effort behind it, operationalizing a teacher-AI complementarity question. It speaks to Teaching and teacher training (Teacher Education), prompting, and the risk of Cognitive Offloading as teachers offload pedagogical decisions to chatbots.

Key Findings

• AI access improved design quality. Bloom attainment (>=2 higher-order tasks) rose from 41.7 percent (no AI) to 91.7 percent (AI access), a significant change (Cochran's Q = 8.86, p = .01, Kendall's W = .37), with a shift toward higher-order Bloom categories (analyze/evaluate/create). ICAP attainment rose modestly (58.3 to 75 percent) but not significantly.
• AI access lowered perceived cognitive effort. The composite effort score dropped sharply from Median 6.33 (no AI) to 3.33 (AI access) (Friedman chi2 = 10.86, p = .004, Kendall's W = .49); all three component measures (pedagogical objectives, task creation, narrative) declined.
• Training did not further improve quality but raised effort slightly. Activity 3 (after training) plateaued on quality yet showed a small, non-significant effort increase — interpreted as possible germane load (deliberate application of new strategies) or extraneous load/fatigue, not resolvable in this design.
• Positive training evaluation and adoption intentions. Post-training evaluations averaged M=4.14/5 (utility highest at 4.25); adoption intentions were strong (M=4.21/6). Pre-training, teachers reported high perceived usefulness (TAM M=6.00/7) but only moderate AI-related technological-pedagogical knowledge (AI-TPK M=3.90/7).
• A caution on interpretation. Quality gains alone do not prove teachers internalized practices — they may reflect delegating routine work to AI or adopting AI-generated structure, consistent with Cognitive Offloading concerns.

What this means for practice

• Faculty developers. Get working AI access into teachers' hands early: access alone raised higher-order Bloom task attainment from 41.7% to 91.7% and dropped the composite cognitive-effort median from 6.33 to 3.33, so a short orientation may deliver more than a gated certification sequence.
• Faculty developers. Plan beyond a single session. A short training did not further improve Learning Design quality and slightly increased effort, so add mentoring, coaching, or longitudinal follow-up to convert stated intentions into changed practice.
• Instructors. Check whether the higher-order thinking in an AI-assisted activity is yours or the chatbot's, since a Bloom-tagged objective can reflect delegated structure rather than internalized design skill and risk Cognitive Offloading.
• Learning designers. Read a small post-training effort increase as possibly productive difficulty rather than failure, and inspect chatbot interaction logs before concluding that the training burdened teachers.
• Researchers. Replicate with larger, more diverse samples and longer consolidation windows to establish whether the quality plateau after training reflects a ceiling effect or the study's short session.

Limitations

• Only 13 higher-education teachers and learning designers at one institution (Danube University, Austria) participated, recruited through an open institutional call — a small, self-selected sample.
• The intervention was a short session, providing no evidence on whether design practices persist or transfer.
• Design quality was scored on a narrow set of manually evaluated binary indicators (Bloom, ICAP, Narrative), which may not capture the broader quality of a learning activity.
• Results reflect one model and interface (iChat) and one training format, and self-reported effort, evaluations, and adoption intentions may not predict actual future use; the authors label the study exploratory.

Citation

Pishtari, G., Gnadlinger, F., & Ley, T. (2026). When Teachers Use AI Chatbots and Are Trained for It: Impact on Learning Design Quality and Cognitive Effort. Artificial Intelligence in Education (AIED 2026), Lecture Notes in Computer Science, Springer, pp. 251-259.