Interpretable Knowledge Tracing via IRT

Two critical gaps in dialogue-based Knowledge Tracing (KT):

Most LLM-based dialogue tutoring systems produce opaque predictions. Huang et al. map raw LLM logits into student ability (θ) and task difficulty (d) parameters via Item Response Theory, yielding predictions that are both more accurate and interpretable in cognitive terms.^Huang Interpretable Knowledge Tracing 2026

The Problem with Opaque KT

Two critical gaps in dialogue-based Knowledge Tracing (KT):

1. Ignored question difficulty: Existing methods rely only on student knowledge states. When a tutor scaffolds up or increases task difficulty, predictions become inaccurate.

2. Opaque representations: High-dimensional LLM embeddings lack interpretability, making it hard to align predictions with learning theory or gain tutor trust.

The Framework

Built on Llama-3.1-8B-Instruct with three modules:

1. Knowledge Estimator

Extracts next-token logits for vocabulary tokens "GOOD" and "BAD" from dialogue history + original question:

θ = z^GOOD − z^BAD

2. Difficulty Estimator

Extracts logits for "HARD" and "EASY" from dialogue history + next tutor-posed task:

d = z^HARD − z^EASY

3. IRT Predictor

Maps ability and difficulty into correctness probability via a 1PL (Rasch) model with learnable scalar α:

p(correct) = 1 / (1 + exp(−α(θ − d)))

"This formulation provides interpretability since both student knowledge and task difficulty are explicitly represented as meaningful cognitive quantities."^Huang Interpretable Knowledge Tracing 2026

Results

Evaluated on QATD2k (real) and MathDial (simulated) tutor-student dialogue datasets:

LLM-based methods substantially outperform traditional deep learning models (∼10-15 AUC points), and the explicit IRT parameterization slightly improves over raw LLMKT.^Huang Interpretable Knowledge Tracing 2026

Relationship to Tutoring-Specific Design

This framework is an enabler for tutoring-specific AI: by explicitly modeling difficulty and student readiness at each turn, the system can dynamically calibrate scaffolds rather than defaulting to full assistance. It operationalizes the insight that general-purpose LLMs can be re-framed as psychometric instruments through prompt design.^Huang Interpretable Knowledge Tracing 2026

Limitations

Connected Concepts

Connected Articles

Citation

Amherst), A.S.H.A.S.J.L.A.L., gaps:, E.D.K.T.S.F.T., simpleKT, D.L.D.D.S.A., & LLMKT, L. (2026). Interpretable Knowledge Tracing via IRT