Synthesis: Semi-structured interviews with 12 teachers who tutored LLM-simulated students (MathDial dataset) reveal key authenticity gaps: overly complex language, lack of emotions, unnatural attentiveness, and logical inconsistency. The study categorizes four real-world student behavior types along scaffolding and presence dimensions, and provides design guidelines for building higher-fidelity LLM student simulations.
Methodology
Martynova et al. interviewed 12 teachers who had extensively interacted with LLM-simulated students during collection of the MathDial dialogue tutoring dataset. The study used a mixed-method approach grounded in two frameworks:
Community of Inquiry (CoI) — capturing social and cognitive presence in learning interactionsScaffolding theory — effective teaching through graduated supportTeachers tutored LLM students in K-12 math problem-solving dialogues, then rated realism and described deviations from authentic student behavior.
Key Findings
Authenticity Gaps in LLM Students
| Issue | Description |
|---|
| Language complexity | Responses too technical, lengthy, and formal for K-12 students |
| Emotional absence | Lack of frustration, fear, embarrassment, or disengagement |
| Unnatural attentiveness | Students too engaged; never lose focus or go silent |
| Logical inconsistency | Knowledge jumps without gradual building; no forgetting |
| No question-asking | Teachers had too much control over discussion flow |
Four Student Behavior Categories
The study classifies real-world student behaviors along two dimensions:
| High Scaffolding Needs | Low Scaffolding Needs |
|---|
| Social Presence | Short/simple writing, negative emotions, disengagement | Asking questions, disagreeing with teacher |
| Cognitive Presence | Gradual knowledge-building, memory/forgetting | Changing tactics based on feedback |
LLMs captured the bottom-right quadrant reasonably well but failed to represent the other three categories.
Design Guidelines
1. Diverse personalities — model Big Five personality traits to produce varied engagement levels and emotional responses
2. Gradual knowledge building — integrate knowledge tracing to avoid unrealistic knowledge jumps
3. Model forgetting — account for memory decay over time
4. Promote question-asking — use context-aware triggers for the LLM student to ask questions
5. Vary language complexity — regulate response length, formality, and introduce age-appropriate errors
6. Allow disengagement — let simulated students lose focus or stay silent, providing authentic teaching challenges
Significance
Teacher training: more realistic LLM student simulations enable scalable practice for pre-service and in-service teachersValidation gap: only 3% of studies simulating learners do post-factum validation — this study provides a framework for itMathDial is the only publicly available dataset of real teacher/LLM-student interactionsAddresses the growing trend of using unvalidated LLM simulations in educational contextsConnected Concepts
K 12Knowledge TracingLLMScaffoldingConnected Articles
Aaai2026 Prompting Literacy K12 — Learning to Use AI for Learning: Teaching Responsible Use of AI Chatbot to K-12 Students Through an AI Literacy ModuleAcademiclaw Student Agent Benchmark — AcademiClaw: When Students Set Challenges for AI AgentsAccess Not Enough AI Tutoring 2026 — Access is Not Enough: Human Support Improves Engagement with AI TutoringAdapt Adaptive Lesson Plan Transformer — AdaPT: Adaptive Lesson Plan Transformer for Cross-Regional and Differentiated InstructionAgent Voice Accents K12 Group Learning — Exploring How Agent Voice Accents Shape Human-AI Collaboration in K-12 Group LearningAgentic AI Education Scoping Review — Agentic AI in Education: A Scoping Review of Research Landscape, Capabilities, and the Frontier Agent ParadigmAgentic AI Pedagogical Best Practice 2026 — Agentic AI and Pedagogical Best Practice: The Tension Between Automation and LearningAgentic Education Coding — Agentic Education with AI Coding AssistantsAgentic Literacy Debt — Agentic Literacy Debt: A Structural Problem the AI Literacy Field Has Not Yet NamedAgents That Teach Incidental Learning — Agents That Teach: Designing Incidental Learning Back into AI-Assisted Software DevelopmentAgreement Not Quality LLM Coding Verification — Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not G...AI Agents Constructive Conflict Design Education 2026 — Enacting Constructive Conflicts with AI Agents to Enhance Reconsideration among Novice Interaction DesignersAI Agents Peer Learning Discourse — When AI Agents Teach Each Other: Discourse Patterns Resembling Peer Learning in the Moltbook CommunityAI Assistance Discretionary Feedback — AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher EducationAI Assisted Learning Modes Eeg — An exploratory behavioral and electroencephalographic study of artificial intelligence-assisted learning modes in hig...AI Availability Student Motivation — Why Put in This Much Effort?": How AI Availability Shapes Students’ Motivation in Introductory ProgrammingAI Campus Wellbeing Tools — AI-Driven Tools for Enhancing Campus Well-being: Prevention and InterventionAI Changing Teaching Workflows — How AI Is Changing Teaching WorkflowsAI Coaching RL Skill Development — AI Coaching for Accelerating Human Skill Development with Reinforcement LearningAI Education Global Capacity — What AI in Education Needs Next: Lessons from Youth Leaders Across Five CountriesAI Enabled Serious Games — AI-Enabled Serious Games: Integrating Intelligence and Adaptivity in Training SystemsAI Engineering Education Balancing Act — Using AI in engineering education: a balancing act, driven by clear purposeAI Generated Traces Novice Programmers — AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional StudyAI In The Wild College — AI in the Wild: A Large Scale Analysis of Authentic Interactions of College Students with Generative AIAI Interlocutor L2 Spoken Dialogue — What Changes When the Interlocutor Is an AI? Interactional Fluency and Linguistic Uptake in L2 Spoken DialogueCitation
Learners?, C.L.E.S.H., Students, T.I.F.T.L., Daheim1,2, D.M.J.M.N., Sachan1, Ö.N.Y.X.Z.M., Fraser, E.Z.T.D.S., many, L.L.M.O., & aims, F.B.H.L.S.A.P.U.T.S. (2026). Can LLMs Effectively Simulate Human Learners? Teachers' Insights from Tutoring LLM Students. Innovative Use of NLP for Building Educational Applications) DOI: https://aclanthology