Evaluating a data-driven redesign process for intelligent tutoring systems

Synthesis: This paper tests whether Huang et al.'s (2021) data-driven redesign process generalizes to instructional units chosen by topic rather than by expected redesign suitability. Applying the process to four units of a middle-school mathematics intelligent tutoring system (MathTutor) — selected precisely because they had already been refined through classroom feedback — the authors refined the knowledge-component model (splitting an unmodeled interpolation skill in Unit 7.05, separating sign-handling skills in the equation-solving units), redesigned tasks and scaffolding, and added adaptive problem selection with mastered-step skipping. In a preregistered crossover study with 123 students, learning gains did not differ between conditions, but the Redesigned Tutor produced more productive time-on-task, more practice opportunities, and greater total knowledge mastered (five additional skills in the same class time). The study is evidence that the redesign method works on content not pre-selected for improvement — and that its benefits are bounded by the amount of practice students actually get.

The core mechanism is making implicit or missing knowledge components explicit: learning-curve analysis in DataShop revealed that Unit 7.05's coordinate-reading problems required an interpolation skill the tutor treated as simple lookup, and equation-solving items conflated negative-term operations (e.g., interpreting "-x" as "-1x") with positive ones. Splitting these KCs and giving them focused, scaffolded practice is what the authors attribute the process-level gains to. The null learning-gain result is interpreted honestly: the post-test included no interpolation items for the newly split skill, and six 20-minute sessions across four units left both conditions below 70% AFM mastery — insufficient practice to reveal downstream gains, a finding that mirrors the mastery learning literature on practice thresholds. The article also reports two notable null results of its own: the redesign did not significantly reduce overpractice beyond 80% mastery, and productivity gains appeared only in Unit 7.06, where interface simplification (relaxed step-order constraints, pre-filled tables) likely reduced idle time.

Key findings

• Learning gains did not differ by condition (Redesigned mean gain 0.20 vs. Original 0.21, t(279) = −0.30, p = .760), despite both being substantial pre-to-post gains (d ≈ 0.57–0.64); the null held across units and interactions (ps > .20).
• Students using the Redesigned Tutor spent significantly more time productively engaged: more logged-in time (47.7 vs. 39.3 min, t(235) = 2.44, p = .015) and more active non-idle time (38.9 vs. 31.6 min, t(234) = 4.62, p < .001), with idle time statistically identical.
• Practice was reallocated toward harder, newly identified skills — practice was significantly more difficult (β = −0.10, p = .020), students received more opportunities overall (β = 0.30, p < .001), and new KCs received relatively more practice (β = 0.23, p = .041), though still less than established KCs (β = −0.74, p < .001).
• Total knowledge mastered increased: iAFM estimates 7.61 vs. 6.03 skills (t(115) = 4.33, p < .001, dp = 0.40) and BKT estimates 8.96 vs. 7.30 (t(115) = 3.67, p < .001) — reflecting five additional skills covered, since average per-KC mastery did not differ (AFM 0.683 vs. 0.658, p = .270).
• The redesign did not significantly reduce overpractice beyond 80% mastery (β = 0.16, p = .080; interaction β = 0.28, p = .083) — the adaptive selection's overpractice-reduction claim was not confirmed in this classroom context.
• Productivity gains were unit-specific: completed steps differed significantly only in Unit 7.06 (β = 37.76, p = .001), the unit whose redesign simplified the interface rather than only the KC model; logged-in time rose 28% there vs. 16% in 7.05, 18% in one equation unit, and −4% in the other.
• Practice quantity likely constrained gains: six 20-minute sessions across four units left AFM mastery below 70% in both conditions, and the post-test lacked items on the newly split interpolation skill — the authors attribute the null primarily to insufficient practice and outcome coverage, not redesign failure.

What this means for practice

• For educational software developers: the method transfers to content nobody pre-selected for improvement. The three-step loop — KC-model refinement from learning curves, task/scaffolding redesign, adaptive problem selection with mastered-step skipping — is implementable with existing DataShop tooling and is the article's practical contribution.
• For evaluation design: pair process measures with post-test items that cover newly split skills, or a genuine improvement will read as a null. The missing interpolation item is the paper's own cautionary example.
• For practitioners interpreting null results: "no learning-gain difference" is not "redesign failed" — check practice dose first. Six 20-minute sessions could not carry either condition to mastery; most deployed tutors may offer less.
• For researchers: the per-KC mastery null alongside the total-mastery gain is a coverage effect, not a depth effect — report both. The overpractice-reduction null (p = .080) is a hypothesis worth re-testing at higher power rather than treating as settled.

Limitations

• A short paper reporting a single classroom study (N = 95 with complete pre/post data, 8 classes, 5 middle schools); process analyses are explicitly exploratory, not preregistered.
• All redesigned content is middle-school mathematics, close to the prior work it generalizes from — domain extension remains untested, as the authors note.
• The crossover design with condition switching after the mid-test introduces possible carryover between tutor versions within students, unaddressed in the paper.
• The redesign's effect on overpractice is inconclusive (marginal p-values, explicitly noted by the authors), so its central adaptive-selection claim rests on reallocation evidence rather than reduction evidence.
• Learning-gain estimates depend on AFM/BKT mastery models applied to both conditions for comparability, which cannot reveal gains on skills the model itself may mis-specify.

Citation

Lyu, Q., Borchers, C., Xia, M., Xiao, K., Carvalho, P. F., Koedinger, K. R., & Aleven, V. (2026). Evaluating a data-driven redesign process for intelligent tutoring systems. In Proceedings of the 27th International Conference on Artificial Intelligence in Education (AIED 2026). arXiv:2603.29094.