🏷️ Concept
Hallucination Risk
Hallucination Risk — the danger that AI systems generate plausible but factually incorrect or fabricated content in educational contexts, where such errors can mislead learners, undermine trust, and produce invalid assessments. Hallucination is particularly consequential in education because students may lack the domain knowledge to detect AI errors, and teachers may rely on AI-generated diagnoses or feedback that appears authoritative but is unfounded.
Hallucination in educational AI takes several forms documented in this wiki's articles: fabricated evidence in student assessment, over-confident misdiagnosis of learner knowledge, and plausible-sounding but incorrect explanations that students accept as truth. The risk is amplified in education because the asymmetry of knowledge between AI and learner means the learner is poorly positioned to verify AI outputs.
Assessment hallucination is particularly damaging. MathCog found that LLMs fabricate evidence quotes not present in student handwriting when diagnosing cognitive skills, with 58.5% of incorrect diagnoses accompanied by false claims of evidential confidence. LLM Fallacy Misattribution documented systematic over-attribution of evidence in LLM reasoning — models claim evidential support where none exists. Both connect to AI Ed Evaluation and Knowledge Tracing concerns about assessment validity.
Tutoring hallucination affects learning directly. LLM Tutoring Feedback Diagnosis Gap found LLMs over-validated incorrect solutions while over-rejecting valid-but-suboptimal reasoning — systemic failures that would mislead both students and teachers. Eduframetrap LLM Sycophancy Educational Safety and Eduguard Safe RAG LLM Tutor address safety mechanisms for educational LLMs. These risks connect to Pedagogical Safety and Human In The Loop AI requirements.
Mitigation approaches include Human In The Loop AI designs where AI supports rather than replaces teacher judgment, evidence-aware architectures that calibrate confidence based on evidential quality (as advocated by MathCog), and RAG-based grounding that constrains LLM outputs to verified sources. The Over Reliance concept is closely related — hallucination is most dangerous when users trust AI outputs uncritically.