Interpretable Knowledge Tracing via IRT

📄 Full text: arXiv:2605.01097 · local

Most LLM-based dialogue tutoring systems produce opaque predictions. Huang et al. map raw LLM logits into student ability (θ) and task difficulty (d) parameters via Item Response Theory, yielding predictions that are both more accurate and interpretable in cognitive terms.^huang-interpretable-knowledge-tracing-2026

The Problem with Opaque KT

Two critical gaps in dialogue-based Knowledge Tracing (KT):

1. Ignored question difficulty: Existing methods rely only on student knowledge states. When a tutor scaffolds up or increases task difficulty, predictions become inaccurate. 2. Opaque representations: High-dimensional LLM embeddings lack interpretability, making it hard to align predictions with learning theory or gain tutor trust.

The Framework

Built on Llama-3.1-8B-Instruct with three modules:

1. Knowledge Estimator

Extracts next-token logits for vocabulary tokens "GOOD" and "BAD" from dialogue history + original question: θ = z^GOOD − z^BAD

2. Difficulty Estimator

Extracts logits for "HARD" and "EASY" from dialogue history + next tutor-posed task: d = z^HARD − z^EASY

3. IRT Predictor

Maps ability and difficulty into correctness probability via a 1PL (Rasch) model with learnable scalar α: p(correct) = 1 / (1 + exp(−α(θ − d)))

"This formulation provides interpretability since both student knowledge and task difficulty are explicitly represented as meaningful cognitive quantities."^huang-interpretable-knowledge-tracing-2026

Results

Evaluated on QATD2k (real) and MathDial (simulated) tutor-student dialogue datasets:

Model	QATD2k AUC	MathDial AUC
DKT	54.73	63.14
DKVMN	51.79	60.64
LLMKT	64.89	75.99
Ours (IRT)	65.25	76.59

LLM-based methods substantially outperform traditional deep learning models (∼10-15 AUC points), and the explicit IRT parameterization slightly improves over raw LLMKT.^huang-interpretable-knowledge-tracing-2026

Relationship to Tutoring-Specific Design

This framework is an enabler for tutoring-specific AI: by explicitly modeling difficulty and student readiness at each turn, the system can dynamically calibrate scaffolds rather than defaulting to full assistance. It operationalizes the insight that general-purpose LLMs can be re-framed as psychometric instruments through prompt design.^huang-interpretable-knowledge-tracing-2026

Limitations

Simulated data: MathDial uses GPT-3.5 as the "student," which may not reflect real learner behavior

Fine-tuning required: LoRA on Llama-3.1-8B; not zero-shot

Binary correctness only: Does not model partial understanding or open-ended reasoning

kt4eqg-personalized-question-generation — 7 of 8 papers in May 28 scan

stanbkt-bayesian-knowledge-tracing — Bayesian BKT framework extends beyond IRT's point-estimate limitations

clara-collaboration-literacy-dashboard — CLARA: An AI-Augmented Analytics Dashboard for Collaboration Literacy

llm-tutoring-feedback-diagnosis-gap — Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

difficulty-aware-dialogue-kt — Difficulty-aware dialogue KT extends IRT-based tracing to conversational settings

reliable-programming-kt — Controlled re-evaluation of PKT protocol reliability

temporal-smoothness-debiased-kt — Doubly robust KT framework correcting selection bias

mbp-kt-meta-behavioral-knowledge-tracing — Meta-behavioral pattern extraction for enhanced collaborative KT

explainable-probabilistic-kt — Probabilistic embeddings replacing deterministic KT vectors

reinforcement-learning-measurement-model-assessment — Reinforcement Learning Measurement Model

cognitive-agent-compilation — CAC makes knowledge tracing explicit and inspectable, complementing IRT-based approaches

ai-tutor-behavioral-evaluation — behavioral evaluation axis for AI tutors — measuring what students actually do with feedback

socratic-ai-dialogue — Modeling readiness for less Socratic, more direct support

affective-tutoring — Combining affective and cognitive learner models

tutoring-specific-vs-general-ai — Tutoring-specific vs. general-purpose AI design

adaptive-learning-systems — Agent-native learner-profile engines like DeepTutor

personalized-learning — Longitudinal memory architectures for individual adaptation

item-response-theory — (create when second source emerges)

llm-student-modeling-memory — Longitudinal memory architecture for student modeling (PersonaVLM framework)

neural-symbolic-knowledge-tracing — Interpretable vs. neural approaches to knowledge tracing

llm-student-misconception-identification — Extends KT beyond correct/incorrect to specific misconception categories- learnopt-exam-cognitive-structure -- Standardized exams have stable latent cognitive structures recoverable via LLM-tagged question analysis and knapsack optimization

llm-item-difficulty-prediction — LLM reasoning traces for item difficulty estimation

Sources

Huang et al. (2026). Interpretable Difficulty-Aware Knowledge Tracing in Tutor-Student Dialogues. arXiv:2605.01097. PDF

codify-socratic-programming-tutor — Competency modeling with topic-level mastery tracking in Codify