When Teachers Use AI Chatbots and Are Trained for It: Impact on Learning Design Quality and Cognitive Effort

Synthesis: In a within-subjects study, 13 higher-education teachers designed learning activities across three conditions — no AI, with a chatbot, and with a chatbot after receiving training on pedagogically grounded AI-interaction strategies. AI access significantly improved design quality (most clearly by raising attainment of higher-order Bloom's tasks) and lowered perceived cognitive effort; the training did not further improve design quality but slightly increased cognitive effort, likely reflecting deliberate application of newly learned strategies. Teachers rated the training as useful and reported strong adoption intentions.

Relevance to AI in Education: This study empirically examines whether and how AI support and guidance shift teachers' Learning Design quality and the cognitive effort behind it, operationalizing a teacher-AI complementarity question. It speaks to Teaching and teacher training (Professional Development), prompting, and the risk of Cognitive Offloading as teachers offload pedagogical decisions to chatbots.

Key Findings

• AI access improved design quality. Bloom attainment (>=2 higher-order tasks) rose from 41.7 percent (no AI) to 91.7 percent (AI access), a significant change (Cochran's Q = 8.86, p = .01, Kendall's W = .37), with a shift toward higher-order Bloom categories (analyze/evaluate/create). ICAP attainment rose modestly (58.3 to 75 percent) but not significantly.
• AI access lowered perceived cognitive effort. The composite effort score dropped sharply from Median 6.33 (no AI) to 3.33 (AI access) (Friedman chi2 = 10.86, p = .004, Kendall's W = .49); all three component measures (pedagogical objectives, task creation, narrative) declined.
• Training did not further improve quality but raised effort slightly. Activity 3 (after training) plateaued on quality yet showed a small, non-significant effort increase — interpreted as possible germane load (deliberate application of new strategies) or extraneous load/fatigue, not resolvable in this design.
• Positive training evaluation and adoption intentions. Post-training evaluations averaged M=4.14/5 (utility highest at 4.25); adoption intentions were strong (M=4.21/6). Pre-training, teachers reported high perceived usefulness (TAM M=6.00/7) but only moderate AI-related technological-pedagogical knowledge (AI-TPK M=3.90/7).
• A caution on interpretation. Quality gains alone do not prove teachers internalized practices — they may reflect delegating routine work to AI or adopting AI-generated structure, consistent with Cognitive Offloading concerns.

Connected Concepts

• Learning Design
• Teaching
• Professional Development
• Educational Development
• Higher Education
• Generative AI
• Prompt Engineering
• Cognitive Offloading

Connected Articles

• Transforming Curriculum Design with Generative AI: A Model for Assessing Teacher Digital Competence — Teacher competence frameworks for GenAI
• A review of intervention designs of LLM Integration in Undergraduate Computer Science Education — LLM interventions in education
• A Systematic Review of Language Educators' Practices and Development with GenAI — Language educators and GenAI
• A cultural-historical activity theory-anchored learning analytics pipeline for early detection and social-epistemic — AI literacy in practice

Citation

Pishtari, G., Gnadlinger, F., & Ley, T. (2026). When Teachers Use AI Chatbots and Are Trained for It: Impact on Learning Design Quality and Cognitive Effort. Artificial Intelligence in Education (AIED 2026), Lecture Notes in Computer Science, Springer, pp. 251-259.