AIEd Knowledge BaseAIEd KB
On this page

Concept

Explainable AI

created 2026-09-07 · updated 2026-10-16

Explainable AI (XAI) in education is the design and study of making an AI system's decisions legible to its educational stakeholders — Learners, teachers, administrators, parents, researchers, and policymakers. The central distinction the field insists on: explaining subject matter (why a fact is true) is not the same as explaining an AI system's decision (why this learner was assigned this activity, why this response was marked incorrect, what evidence supports a risk prediction). Education brings distinctive explainability needs — noisy learning data, explanations that can directly support Metacognition and Self-Regulated Learning, and stakeholders who require fundamentally different explanation types. The operative design question is explanation quality, not mere explanation availability: an explanation that is technically present but unreadable, misleading, or misaligned to its audience can do more harm than no explanation at all.

Questions to Consider

  • When an AI tutor tells you why a hint was given, is that explaining the subject matter or explaining the system's decision? Can you name three examples of each in your own use of educational AI?
  • Who needs explanations in education — and do learners, teachers, and policymakers need the same kind? What would each use an explanation for?
  • An AI flags a student as at-risk for dropping out. What does a teacher need to know to act on that, versus what the student needs to know? Is the same explanation appropriate for both?
  • The page argues explanation quality matters more than explanation availability. What makes a technically present explanation fail — can you think of a time an explanation was there but useless, or worse, misleading?
  • Trust and explanation are linked but not identical. Why might a confident, fluent explanation create false confidence in a flawed system — and how would you detect that happening?

Introduction

Explainable AI in education names the growing expectation that AI systems in classrooms should not be black boxes. Because AI in education affects consequential decisions — grades, risk flags, learning paths, resource recommendations — stakeholders increasingly demand to know not just what the system concluded but why. Education sharpens this into two distinct questions: explaining the subject matter being learned, and explaining the AI system's own decision-making. Conflating them is a category error with practical consequences: an AI that explains a physics answer perfectly still gives a student and teacher no insight into why the system ranked them at-risk, recommended a certain activity, or marked a response incorrect.

Explaining subject matter vs. explaining the system's decision

The field's founding contribution — the XAI-ED framework (Khosravi et al., 2022) — insists education has distinctive explainability needs beyond general-purpose XAI. Chief among them is the split between two explanation targets:

  • Subject-matter explanations clarify content: why a hint addresses a misconception, why an answer is incorrect, how a physics result follows from principles. These are pedagogical explanations that support Scaffolding, Feedback, and Metacognition.
  • System-decision explanations clarify the model: why this learner was assigned this activity, why the system predicts this student is at risk, what evidence supports a knowledge-tracing or Learning Analytics prediction. These are transparency explanations that support Trust Calibration, Bias Mitigation, and accountability.

The distinction matters because they serve different stakeholders and different purposes. A learner answering "why is this marked wrong?" mostly needs the subject-matter explanation; a teacher deciding whether to act on a risk flag, or a policymaker auditing for bias, needs the system-decision explanation. Designing a single explanation that serves both is rarely possible — which is why multi-stakeholder design is a core XAI-ED theme.

Who needs explanations: multi-stakeholder design

  • Learners need explanations that support their own learning and self-regulation — why a hint was given, why their answer was marked incorrect, why this resource is recommended (supporting Self-Regulated Learning). The student-perspective evidence shows learners draw a sharp line between accepting AI feedback (useful for revision) and ceding grading authority (reserved for the human instructor) — a calibrated, function-matched stance activated by transparency about AI involvement. Value-sensitive design work with community college students sharpens the point: students preferred collaborative, humanized explanations (e.g., "the AI might be uncertain here, so let's check this together") over raw model confidence or technical transparency, because transparency alone has little value unless it directly supports their learning. The study surfaced a transparency-vs.-interpretability tension that pushes explanation design toward learner-facing semantics rather than feature-importance output.
  • Teachers need explanations that inform intervention — which students are at risk and why, on what evidence. Explainability studies with teachers show domain-specific, curricular-language explanations build acceptance and calibrated trust more effectively than generic feature-importance ones, yet teachers still want real classroom experience before full reliance — explanation alone does not confer calibration.
  • Developers and researchers need explanations to debug model behavior and detect bias — surfacing which features drive predictions.
  • Administrators and policymakers need explanations for accountability, Privacy, and AI Regulation in Education compliance (e.g. the right to explanation), and to audit whether AI-driven decisions are fair and equitable.

Approaches and formats

The XAI-ED framework catalogs the main explanation modalities: visual (heatmaps, decision trees), textual (natural-language justifications), example-based (counterfactuals, nearest neighbors), feature-importance rankings, rule extraction, and model simplification. It also maps approaches to model classes:

  • White-box models (decision trees, linear models, rule-based) are inherently interpretable.
  • Black-box models (neural networks, ensembles) require post-hoc explanation methods.
  • Glass-box approaches try to balance accuracy with transparency.

The concrete AIED evidence base spans all of these. Interpretable knowledge tracing makes learner-knowledge models inspectable directly (Interpretable Knowledge Tracing, Explainable Knowledge Tracing via Probabilistic Embeddings and Pattern-based Reasoning, Neural-Symbolic Knowledge Tracing: Injecting Educational Knowledge into Deep Learning for Responsible Learner Modelling). Self-explaining surrogates distill a black-box model into a small, interpretable language model for Learning Analytics (Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics). Counterfactual explanations — "what would need to change for a different outcome" — support educational decision support and recourse (From Student Risk Prediction to SC2R: Semantics-Constrained Counterfactual Recourse for Educational Decision Support). Federated + explainable learning analytics shows explanation quality can drift (calibration degrades) even when ranking stability holds, underlining that explanations are not a fixed property but a system output to be measured (Federated and Explainable Learning Analytics for Privacy-Preserving Academic Risk Modeling Across Heterogeneous Educational Institutions). And interpretable affective ITS demonstrates explanations in emotion-aware tutoring (An Interpretable Closed-Loop Intelligent Tutoring System for Multimodal Affective Feedback in Asynchronous Presentation Training).

Explanation quality, not availability

A recurring lesson across the evidence: having an explanation is not enough; the explanation must be right for its audience, accurate, and calibrated to stakes. The XAI-ED framework names the pitfalls explicitly:

  • Explanation overload — too much information overwhelms the user and negates the benefit.
  • Misleading explanations — post-hoc explanations may not reflect the model's actual reasoning, giving false confidence.
  • Confirmation bias — users selectively attend to explanations that confirm existing beliefs.
  • Over-trust — fluent explanations can create false confidence in flawed systems, feeding over-reliance (the obverse of Trust Calibration).
  • Gaming the system — students may exploit explanations to circumvent actual learning.

Explanation quality also has an equity dimension: an explanation that is technically present but unreadable to a given stakeholder — or that obscures the bias in a prediction — fails its purpose. This is why the design question is quality and fit, and why human-centered, stakeholder-specific explanation design is inseparable from the technical generation of explanations. Effective XAI is a communication act designed for the recipient's cognitive needs, not merely a technical artifact.

Explanation is not always a leveler. In a 2 × 2 vignette experiment with 250 seventh-grade students, a written rationale for a mathematics score raised acceptance and perceived fairness in both conditions but widened rather than closed the gap between teacher- and AI-made decisions (Zhang et al. (2026)).

Citations behave as a credibility cue rather than evidence of correctness: attaching citations and authority signals to a debate interface raised abandonment of a correct judgment for the AI's position from 15.0% to 31.8%, and cited cases were sometimes misattributed or fictitious (Tuo et al., 2026).

Two cautions sharpen this further, both of which the wiki's newest contribution on the subject makes central. First, the explanation machinery is not itself neutral: post-hoc methods such as LIME and SHAP can be unfaithful to the model's actual behavior, so a technically present explanation may mislead rather than inform (Lund et al. 2026, drawing on Chuan et al. 2024). Second, explanation is not accountability. An account of which features drove a prediction does not reveal whether those features were appropriate to use, whether the training data was representative, or whether the system's design reflected sound judgment; explanations can create the appearance of transparency while leaving the structural conditions that produced a decision untouched (Mittelstadt et al. 2019). For education this means the question to keep asking is not whether an explanation was produced but whether the person receiving it — a student, a teacher, an advisor — could understand it, act on it, or contest the decision behind it. The same failure of legibility appears on the security side of automated assessment: Humble's (2026) red-team of an AI grading tool found it silently disabling the chat after blocking a prompt injection, and — having announced it would never follow embedded instructions — following them in six further runs on the same file, leaving the user no reliable signal on which to base reliance.

Explainable-by-design is one answer to the post-hoc faithfulness problem. Li, Yang and Fang (2025) parameterize an explanation decoder by the same fused representation and predicted score that decide the Assessment, so that a low score on formative questioning yields a rationale naming insufficient probing questions, and pair it with dual-lens attention over curriculum standards and subject-specific rubric moves. Attention-to-rubric alignment reaches 78.0% against 41.7% for GPT-4 zero-shot and 32.1% for BERT, and faithfulness is probed by counterfactual deletion of rubric-critical spans alongside human ratings on a rubric-anchored checklist, giving an explanation-credibility score of 0.78 — an increase of 0.31 over BERT-base. The audit also shows where the architectural claim thins: on emotional cues the model allocates 28.4% of attention weight against an expert 15.2% (alignment 0.53), with one failure case assigning 28% to the token "frustrated", which the authors read as overfitting to affect rather than pedagogy and name as an area for refinement. Embedding explanations in the decision path makes them more faithful than post-hoc rationales; it does not make them correct.

Teaching explainability as accountability practice

If explanation quality decides whether XAI is useful, then producing explanations has to be taught as a professional habit rather than demonstrated as a capability. Lund and colleagues (2026) propose doing this across four pillars — answerability (the obligation to give reasons to those affected), responsibility (harm anticipated across the lifecycle, not defended after the fact), enforcement (consequences inside the course) and reflexivity (documented examination of one's own assumptions) — each with its own assignments and its own classroom cost.

For explainability specifically, the assignments that matter are the ones that force explanation out of the notebook: graded model cards weighted alongside accuracy metrics, and structured explanation audits in which students apply interpretability tools to their own models and then present the results to an audience without a shared technical background. Enforcement is the pillar most often missing from ethics-adjacent courses and the one that makes the rest more than symbolic — rubrics that reward responsible documentation, projects that can be returned for revision on ethical grounds, and peer review conducted against accountability criteria rather than technical ones alone. The paper is candid that the tools differ sharply in cost: model cards and positionality statements need no new software and risk only superficial compliance, whereas peer panels and stakeholder engagement require coordination and institutional buy-in, which is why it recommends a staged adoption rather than an all-or-nothing commitment. See Curriculum Design for where these fit in a program.

Connected Concepts

Connected Articles

Citation

Khosravi, H., Buckingham Shum, S., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable Artificial Intelligence in education. Computers and Education: Artificial Intelligence, 100074.

Metadata

Intended audienceLearners, Researchers, Instructional Designers, Instructors
Learning and instructionMetacognition
Technologies and techniquesHuman-in-the-Loop, Intelligent Tutoring, Learning Analytics, Learner Modeling and Adaptive Instruction
Assessment and measurementAutomated Assessment
Equity, ethics, and responsible useBias Mitigation, Trust Calibration, Pedagogical Safety
Foundations of AI in educationAI Literacy
SourceHistoryEditChat

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.