🧠 AI Ed Wiki

📄 Research Article

Learning to Use AI for Learning: Teaching Responsible Use of AI Chatbot to K-12 Students Through an AI Literacy Module

● high · 2026-08-06ai-literacy, prompting-literacy, k-12, llm, ai-education, scenario-based-learning, automated-grading, student-experience, rag

Synthesis: An LLM-based interactive module teaches K-12 students prompting literacy through scenario-based deliberate practice with an AI auto-grader providing immediate, detailed feedback. Deployed across 11 secondary classrooms in two iterations, the module improved students' prompting skills (particularly embedding background context) and confidence in using AI for learning. The study also validates an AI-based auto-grader achieving 0.92 average accuracy and identifies True/False + open-ended questions as more effective than MCQs for assessing prompting literacy.

Study Design

Xiao et al. designed and deployed a web-based interactive instructional system to teach prompting literacy to secondary-education students. The module was grounded in two learning sciences principles: learning-by-doing and elaborated immediate feedback. Students practiced prompt writing in three hypothetical learning scenarios (biology, geography, math), each paired with a unique instructional activity (extending knowledge, quiz preparation, homework struggle).

After a student writes a prompt, an LLM-based auto-grader (GPT-4o) evaluates it across preset dimensions and delivers immediate, detailed feedback. The pipeline mirrors authentic AI chatbot interaction: write prompt → receive AI response → get graded feedback.

Two iterations were conducted across 11 secondary classrooms:

  • Study 1 (June 2024): 111 students, 6 classrooms in East Asia
  • Study 2: Assessment iteration follow-up with similar population

    • The Auto-Grader

    The AI auto-grader achieved 0.92 average accuracy across dimensions when grading student-written prompts, using human labels as ground truth:

    DimensionAccuracy
    Relevance0.98
    Background/Context0.96
    Conciseness0.93
    Elaboration0.90
    No Direct Answer0.88
    Clarity of Purpose0.85

    The lowest accuracy (Purpose, 0.85) stemmed from the auto-grader over-generating keywords or conflating Purpose with No Direct Answer criteria. The auto-grader tended to weigh heavily on some keywords while ignoring others — a known limitation of LLM-based grading.

    Key Findings

    Prompting Skill Improvement

  • Students improved significantly at embedding background/context information in prompts (McNemar test, p = .039 from Q1 to Q3)
  • Students performed well on Relevance, Conciseness, and Purpose even in the first question (ceiling effects)
  • Prior AI usage frequency was positively correlated with initial prompt quality (r = 0.27, p = .017), suggesting an equity concern

    • Confidence and Perception

  • Self-reported confidence in using AI for learning increased by 10.4% (p < .001)
  • 87% of students reported learning AI-related knowledge (how to use AI for learning, how to ask effective questions, AI's capabilities)
  • Students valued: direct AI interaction, scenario-based design, immediate comprehensive feedback, and visual elements

    • Assessment Design Lessons

  • MCQs suffered from ceiling effects — students could identify good prompts conceptually but couldn't write them effectively
  • True/False + open-ended questions demonstrated better item difficulty and discrimination than MCQs
  • None of the original MCQ items fell into the desired difficulty range [0.3, 0.7], while 60% of OE and 30% of TF questions did

    • Challenges Identified

  • Productive struggles: difficulty writing effective prompts (the core skill being taught)
  • Extraneous load: slow AI response times, login issues, limited typing skills (22 students reported this)
  • Scenario variety: some students wanted non-STEM scenarios
  • LLM response latency disrupted the practice flow

    • Design Implications

    The study demonstrates that Prompt Engineering can be taught effectively to K-12 students through structured practice with automated feedback. Key design principles:

    1. Scenario-based deliberate practice with authentic AI interaction

    2. Immediate, dimension-level feedback powered by LLM auto-grading

    3. Assessment aligned to competency — open-ended + T/F outperform MCQs for higher-order prompting skills

    4. Addressing the digital divide — prior AI access correlates with initial performance, underscoring the need for in-school prompting literacy instruction

    Connected Concepts

  • AI Education
  • AI Literacy
  • Automated Grading
  • K 12
  • LLM
  • Prompt Engineering
  • Student Experience
  • RAG

    • Connected Articles

  • Academiclaw Student Agent Benchmark — AcademiClaw: When Students Set Challenges for AI Agents
  • Access Not Enough AI Tutoring 2026 — Access is Not Enough: Human Support Improves Engagement with AI Tutoring
  • Adapt Adaptive Lesson Plan Transformer — AdaPT: Adaptive Lesson Plan Transformer for Cross-Regional and Differentiated Instruction
  • Affective Text Wearable Student Health — A Formative Study of Brief Affective Text as a Complement to Wearable Sensing for Longitudinal Student Health Monitoring
  • Agency Gap AI Writing — The agency gap in AI-supported writing: how reactive and proactive agent designs shape multimodal reasoning
  • Agent Voice Accents K12 Group Learning — Exploring How Agent Voice Accents Shape Human-AI Collaboration in K-12 Group Learning
  • Agentic AI Education Scoping Review — Agentic AI in Education: A Scoping Review of Research Landscape, Capabilities, and the Frontier Agent Paradigm
  • Agentic Literacy Debt — Agentic Literacy Debt: A Structural Problem the AI Literacy Field Has Not Yet Named
  • Agentic Workflows Education — Agentic Workflows in Education
  • Agents That Teach Incidental Learning — Agents That Teach: Designing Incidental Learning Back into AI-Assisted Software Development
  • Agreement Not Quality LLM Coding Verification — Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not G...
  • AI Adoption Training Public Sector — The Main Barrier to AI Adoption in the Public Sector is Lack of Training
  • AI Adult Learning Guidelines Dis2026 — Guidelines for Designing AI Technologies to Support Adult Learning
  • AI Agents Peer Learning Discourse — When AI Agents Teach Each Other: Discourse Patterns Resembling Peer Learning in the Moltbook Community
  • AI Assessment Human Tutors — AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice
  • AI Assessment Scale Reform — A bit of chaos and madness": The AI Assessment Scale and the work of assessment reform
  • AI Assistance Discretionary Feedback — AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education
  • AI Assisted Learning Modes Eeg — An exploratory behavioral and electroencephalographic study of artificial intelligence-assisted learning modes in hig...
  • AI Assisted Se Curriculum Syllabus Analysis 2026 — Mapping the Emerging Curriculum for AI-Assisted Software Engineering via Syllabus Analysis
  • AI Availability Student Motivation — Why Put in This Much Effort?": How AI Availability Shapes Students’ Motivation in Introductory Programming
  • AI Campus Wellbeing Tools — AI-Driven Tools for Enhancing Campus Well-being: Prevention and Intervention
  • AI Changing Teaching Workflows — How AI Is Changing Teaching Workflows
  • AI Education Global Capacity — What AI in Education Needs Next: Lessons from Youth Leaders Across Five Countries
  • AI Enabled Serious Games — AI-Enabled Serious Games: Integrating Intelligence and Adaptivity in Training Systems
  • AI Engineering Education Balancing Act — Using AI in engineering education: a balancing act, driven by clear purpose

    • Citation

    Xiao, R., Hou, X., Tseng, Y.-J., Nieu, H., Liao, G., Stamper, J., & Koedinger, K. R. (2026). Learning to Use AI for Learning: Teaching Responsible Use of AI Chatbot to K-12 Students Through an AI Literacy Module. AAAI.