StanBKT: Rethinking Parameter Estimation in Bayesian Knowledge Tracing

Synthesis: StanBKT is an Open Source Python package that reimplements Bayesian Knowledge Tracing in Stan, replacing the expectation-maximization point estimates that most BKT tools produce with full posterior inference. Pradhan et al. (2026) evaluate it on two datasets and reach a result that is easy to misread: on the large ASSISTments 2020 dataset every inference method performed identically on prediction — accuracy 0.694, AUC 0.711, RMSE 0.449 — so the choice is purely about cost and uncertainty, ranging from 822 seconds for Hamiltonian Monte Carlo to 6.83 seconds for MAP estimation. The payoff is demonstrated on a separate randomized experiment, where posterior contrasts let the authors distinguish which perceptual-cue effects on learning, forgetting, guessing, and slipping were credible and which were indistinguishable from noise.

Overview

BKT is one of the oldest and most interpretable student-modeling approaches in intelligent tutoring systems and educational data mining. Its standard implementations, however, estimate four parameters — learn (P(T)), forget (P(F)), guess (P(G)), and slip (P(S)) — as single point estimates by expectation-maximization or similar optimization. That has a direct practical consequence the authors single out: with only a point estimate, a researcher cannot say whether the difference between two conditions is real, which makes adaptive intervention comparisons and A/B tests harder to interpret than they need to be.

StanBKT's contribution is to move the whole estimation problem into a probabilistic programming framework while keeping the classical hidden Markov structure of BKT intact — the model stays interpretable, but the parameters come with distributions attached.

Study Design & Method

• Package scope. A unified interface over four inference methods — Hamiltonian Monte Carlo (HMC), variational inference, Pathfinder, and optimization-based MAP estimation — plus standard, grouped, and hierarchical BKT formulations, flexible prior specification, posterior predictive inference, and utilities for visualization and diagnostics (e.g. trace plots for assessing mixing and convergence).
• Scalability evaluation. The ASSISTments 2020 dataset (student and teacher interactions from the 2019-2020 school year, OSF osf.io/q7zc5), restricted to the 10 knowledge components with the most interactions and to student sequences with at least 10 observations, because the full dataset contains several very sparsely observed knowledge components. The standard BKT model with a forgetting parameter was fit with each method and compared against the pyBKT package, which uses expectation-maximization.
• Predictive metrics. Classification accuracy at a 0.5 threshold, area under the ROC curve, and RMSE, capturing classification, ranking, and probability quality respectively. Predictions used posterior point estimates (posterior means) for the Bayesian methods.
• Compute. A single node with 40 allocated CPU cores from dual AMD EPYC 9654 processors and 64 GB RAM; MCMC ran four parallel chains with 1,000 warmup and 1,000 sampling iterations each. The authors note runtimes are not strictly normalized across methods, since MCMC produces thousands of draws.
• Experimental evaluation. An open-source perceptual-cue intervention study (OSF osf.io/d3tzq; Ottmar et al., 2025): originally 1,110 US sixth-grade students, restricted to the 688 who completed all required tasks and practice items, solving 29 order-of-operations problems for 19,952 problem-solving observations. Participants were randomly assigned to one of nine between-subjects conditions in a 3 × 3 design crossing spacing and color cues (congruent, incongruent, neutral; roughly 70-80 students per condition).
• Analysis approach. The MultiBKT model estimated condition-specific parameters; the marginal effect of each congruent/incongruent cue was computed as a difference against the neutral condition, summarized by posterior median and 95% credible interval, with intervals excluding zero treated as evidence of a systematic change.

Key Findings

• Prediction is identical across inference methods. On ASSISTments 2020, accuracy (0.694), AUC (0.711), and RMSE (0.449) differed only in the decimals between pyBKT/EM and every StanBKT method. The taxonomy of choice is therefore cost, not accuracy: pyBKT 3.35 s, MAP 6.83 s, variational inference 26.8 s, Pathfinder 151 s, and MCMC 822 s for full posterior inference.
• Full posterior inference is affordable but expensive, and the shortcuts are not free conceptually. Variational inference and Pathfinder remain substantially faster than MCMC and, per the package documentation the paper summarizes, offer approximate posterior summaries — Pathfinder generally capturing posterior uncertainty more faithfully than standard variational inference — while MAP collapses the posterior to a single mode and quantifies no uncertainty at all.
• Condition-specific contrasts become the actual analysis. In the perceptual-cue experiment, no credible effects appeared for the learning rate P(T): estimates sat near zero with credible intervals spanning zero for both spacing and color cues. Most effects on forgetting P(F) and slipping P(S) were weak or uncertain.
• Only a few effects survived the credible-interval test. The reliable deviations were a small reduction in forgetting under incongruent color (−0.012, 95% CrI [−0.023, −0.003]) and a modest reduction in slipping under congruent spacing (−0.030, 95% CrI [−0.059, −0.001]).
• The clearest signal was in guessing, not learning. Both cues produced sizable credible deviations in P(G), particularly under incongruent conditions (spacing −0.058 [−0.093, −0.024]; color −0.082 [−0.121, −0.040]), consistent with reduced guessing under more structured or attention-demanding conditions.
• The headline interpretation is a caution, not a win. The authors read the pattern as spacing and color cues being associated mainly with changes in response behavior (guessing and slipping) rather than with learning dynamics, and they note there is little evidence of changes in learning rates that would indicate enhanced learning — a conclusion that a point-estimate analysis could not have drawn as confidently.
• Simulated recovery supports the machinery. In a simulated example, StanBKT recovered the true data-generating parameters, giving a check that the Bayesian implementation estimates what it claims to.

What this means for practice

• Researchers. Report credible intervals whenever you compare conditions: in the perceptual-cue experiment most effects on forgetting P(F) and slipping P(S) were weak or uncertain, and the only reliable deviations were a small reduction in forgetting under incongruent color (−0.012, 95% CrI [−0.023, −0.003]) and a modest reduction in slipping under congruent spacing (−0.030, 95% CrI [−0.059, −0.001]).
• Choose the inference method from the uncertainty you need, not from accuracy: MAP at 6.83 s is a drop-in replacement for pyBKT when posterior uncertainty is not needed, while MCMC at 822 s buys full posterior inference for mastery trajectories and condition comparisons.
• Use the grouped or hierarchical BKT formulations when skills or students are sparsely observed, so estimates pool information instead of resting on unstable per-group point estimates.
• Read cue effects on guessing and slipping as changes in response behavior rather than learning gains — both cues produced sizable credible deviations in P(G) (spacing −0.058 [−0.093, −0.024]; color −0.082 [−0.121, −0.040]) while no credible effect appeared for the learning rate P(T).
• Developers. Ship posterior predictive checks and trace plots alongside the parameter estimates, so practitioners can judge mixing and convergence before acting on a mastery model, and treat StanBKT as a complement to neural or neural-IRT alternatives rather than a replacement for them.

Limitations

• Runtime comparisons are not normalized across methods, since MCMC stores full parameter trajectories across four parallel chains with 1,000 warmup and 1,000 sampling iterations each.
• The scalability study is restricted to the 10 knowledge components with the most interactions in ASSISTments 2020 and to student sequences with at least 10 observations, leaving sparsely observed skills untested at scale.
• The faster alternatives carry approximation error — MAP collapses the posterior to a single mode and quantifies no uncertainty at all, and variational inference and Pathfinder may underestimate posterior uncertainty in complex or weakly identified models.
• Better prediction is not on offer: accuracy (0.694), AUC (0.711), and RMSE (0.449) were identical between pyBKT/EM and every StanBKT method, so a switch to Bayesian inference buys inference quality rather than fit.

Citation

Siddhartha Pradhan, Yanping Pei, Morgan Lee, Puyuan Zhang, Erin Ottmar, Adam C. Sales (2026). StanBKT: Rethinking Parameter Estimation in Bayesian Knowledge Tracing. arXiv preprint (cs.HC, cs.CY, stat.AP, stat.ME).