Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols

Synthesis: > This ITS 2026 study re-evaluates attention-augmented Programming Knowledge Tracing (PKT) models on the CodeWorkout dataset and argues that many reported performance gains are artifacts of flawed experimental protocols rather than genuine architectural superiority. The authors identify three systemic biases in prior work — a causally invalid Softmax dimension in attention-based code representation, temporal causality violations from ignoring ServerTimestamp during sequence construction, and hyperparameter leakage across cross-validation folds — and show that once these are rectified, the performance gap between attention-enhanced models and the standard DKT baseline narrows dramatically, with architectural complexity failing to translate consistently into better prediction.

Key Findings

1. Attention-augmented PKT models (Code-DKT, ECKT) report gains over DKT largely because of causal flaws in their evaluation: the Softmax in attention-based code representation normalizes across the time dimension (dim=1) rather than the path dimension (dim=2), introducing look-ahead leakage.
2. Failing to sort student attempts by ServerTimestamp violates temporal causality — a "submit-and-fix" cycle can place a later failure before an earlier success in the input sequence, systematically overestimating performance.
3. Under a controlled protocol with task-specific hyperparameter optimization, the performance gap between attention-enhanced models and standard DKT is significantly reduced; DKT on assignment 439 improves from 0.6821 to 0.7480 AUC purely from proper tuning.
4. Extending the maximum sequence length from 50 to 100 does not produce consistent gains, suggesting longer interaction histories add noise rather than useful context.

Analysis

This ITS 2026 paper by Jaewook Kim and Hyeoncheol Kim challenges claims about attention-augmented Programming Knowledge Tracing (PKT) superiority. The authors identify three critical protocol flaws in the official Open Source implementation of Code-DKT: attention dimension misconfiguration (Softmax applied across the temporal axis, leaking future information into code-feature weights), temporal causality violations (ignoring ServerTimestamp and aggregating attempts by raw CSV order), and hyperparameter leakage across cross-validation folds. When controlled, the gap between attention-enhanced models and standard DKT narrows significantly.

The study reinforces Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education by showing that experimental protocol flaws can produce misleading performance claims in AI Ed Evaluation. It also carries lessons for Benchmark design in educational AI, arguing for standardized, causally valid evaluation frameworks. The CodeWorkout dataset protocol serves as a template for Programming Intelligent Tutoring Systems research. The paper connects to a broader re-analysis culture in Knowledge Tracing where streamlined architectures (e.g., simpleKT, ReKT) challenge the value of added complexity.

Reliability Issues in Prior PKT Evaluations

The paper's audit of Code-DKT's official implementation surfaces three distinct sources of bias. First, the attention mechanism that weights code paths concatenates each path's node embeddings with the student's correctness vector and computes attention over the entire interaction sequence — normalizing Softmax across time (dim=1) lets the importance of a code feature at time step t be influenced by future states, a direct violation of the causal principle that predictions must be conditioned only on current and historical data. Correcting this to normalize across the path dimension (dim=2) removes the artificial inflation.

Second, sequence construction is chronologically unsorted. The codebase aggregates interactions by raw appearance order rather than by ServerTimestamp, so in rapid debugging cycles a later failure attempt can precede an earlier success in the input. This introduces look-ahead bias and explicit data leakage into Learner Modeling and Adaptive Instruction, obscuring the true latent learning process. Third, hyperparameter sensitivity is substantial and task-dependent: optimal configurations vary not only across models but across programming assignments within a model, driven by differences in assignment difficulty, interaction length, and response distributions. Adopting a single global hyperparameter configuration biases comparative evaluation, especially against baseline architectures.

A Controlled Evaluation Protocol

To address these flaws, the authors propose three protocols. For causal integrity, they rectify the attention computation by normalizing across the path axis. For chronological alignment, they enforce a strict ascending sort by ServerTimestamp as a preprocessing step, reconstructing actual learning trajectories without relying on physical storage order. For reproducibility, they adopt an assignment-wise tuning strategy: hyperparameters are selected via grid search on a single designated fold (fold 0) and then fixed uniformly across all remaining folds, with independent fixed random seeds governing train/test splits, cross-validation folds, and model initialization.

Analyzing the CodeWorkout dataset (69,627 interactions from 413 students across five assignments), they find that the 95th percentile of attempt-sequence lengths ranges from 81 to 135 across assignments — well beyond the conventional Lmax = 50 truncation used by Code-DKT. This motivates a systematic comparison of Lmax = 50 versus Lmax = 100 to determine whether preserving longer interaction histories improves modeling.

Results

Under the corrected evaluation pipeline, chronological alignment produces inconsistent performance shifts: while DKT's AUC on assignment 439 rises slightly (0.6768 → 0.6821), Code-DKT drops noticeably on several datasets (e.g., 0.7445 → 0.7116 on 439), confirming that prior high scores were partly inflated by future-information leakage. Rectifying the attention computation does not uniformly help either — on several assignments the rectified Code-DKT underperforms the tuned DKT baseline despite using code-level representations, suggesting code2vec-style embeddings do not consistently pay off under causally valid settings.

Task-specific hyperparameter optimization is a prerequisite for fair comparison. DKT improves from 0.6821 to 0.7480 AUC on assignment 439 purely from tuning, and optimal early-stopped epochs vary from 42.4 to 75.2 across assignments. ECKT proves more stable than Code-DKT, consistently matching or outperforming DKT and narrowing gaps where Code-DKT struggles, suggesting that structured integration of programming context is more robust than simple feature concatenation. Restoring the W0 projection component yields only marginal changes. Extending Lmax to 100 produces mostly degradation or negligible change across models, indicating that longer sequences introduce irrelevant or weakly informative interactions.

What this means for practice

• Software developers. Normalize the attention weights in code representation across the path dimension (dim=2) rather than the temporal dimension (dim=1) before reusing Code-DKT: normalizing over time lets a code feature's importance at time step t depend on future states, which inflates reported AUC.
• Sort every attempt sequence by ServerTimestamp as an explicit preprocessing step — in a submit-and-fix debugging cycle the raw file order can place a later failure before an earlier success and leak future information into the input.
• Tune hyperparameters per assignment instead of once globally: optimal early-stopped epochs ranged from 42.4 to 75.2 across assignments, and DKT's AUC on assignment 439 rose from 0.6821 to 0.7480 purely from task-specific tuning.
• Keep the truncation at Lmax = 50 rather than extending it to 100: the longer setting produced mostly degradation or negligible change, suggesting extended histories add irrelevant or weakly informative interactions.
• Benchmark against a properly tuned baseline before claiming architectural superiority, and prefer structured context integration (ECKT) over simple feature concatenation (Code-DKT), which underperformed the tuned DKT baseline on several assignments under causally valid settings.

Limitations

• The re-evaluation covers a single dataset — CodeWorkout, 69,627 interactions from 413 students across five assignments — and a subset of PKT models (DKT, Code-DKT, ECKT); the authors state that extending the rectified protocol to broader domains and architectures remains future work, so the corrected rankings are not yet shown to generalize.
• Model comparisons rest on one 80/20 train/test split with five cross-validation folds, and hyperparameters were chosen by grid search on a single designated fold and then fixed across all folds, so the reported ordering is conditional on that one fold's selection.
• The sequence-length finding rests on only two settings, Lmax = 50 and Lmax = 100, so intermediate truncations and other architecture families remain untested.

Citation

Kim & Kim (2026). Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols. ITS 2026 (Springer LNCS).