Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S.

Synthesis: Leite, Zhang, Rana, Hao, Hatch, Kong and Kuang (2025) pooled 18 experimental studies, 77 effect sizes and 11 intelligent tutoring systems conducted in U.S. K-12 schools, and found a significant but modest positive effect on achievement (g = 0.271, SE = 0.011, p = 0.001) — smaller than most earlier meta-analyses because this review restricted itself to experimental designs in one national system. The headline contribution is not the average but the heterogeneity: effects were stable across elementary and middle school and across low-achieving, English-learner and low-income populations, yet dropped sharply in studies that included rural schools (g = 0.146) and in implementations lasting longer than six months (g = 0.110, not significant). A MetaForest analysis ranked worked-out examples as the single most important moderator, with on-demand hints also significant while just-in-time hints were not — evidence for student-controlled support over system-triggered interruption. Only 8 of the 18 studies reported differential attrition at all, and just five met What Works Clearinghouse standards without reservations, so the authors read the corpus as a reporting problem as much as an evidence problem.

Key Findings

1. The average effect is real but modest: g = 0.271. Eighteen experimental studies of 11 systems produced a significant positive pooled effect on achievement, well below the effects reported by earlier Intelligent Tutoring reviews that also admitted quasi-experimental and observational designs.
2. Grade level and learner disadvantage did not moderate effects. Elementary (g = 0.266) and middle school (g = 0.256) were nearly identical, and low achievers (g = 0.270), English learners (g = 0.266) and students in poverty (g = 0.303) showed no disadvantage — contrary to earlier warnings that tutoring systems widen gaps.
3. Rural settings showed roughly half the effect. Studies including rural schools produced g = 0.146 against g = 0.298 for urban and g = 0.260 for suburban settings, a gap the authors attribute to infrastructure limits rather than to the systems themselves.
4. Student-controlled support outperformed system-triggered support. Worked-out examples (g = 0.324) and on-demand hints (g = 0.316) were significant, while just-in-time hints (g = 0.337) and video sequences were not — the only feature with permutation importance above the 50th percentile was worked-out examples.
5. Duration mattered in a non-linear way. Effects peaked at 2 weeks to 3 months (g = 0.449) and fell to g = 0.225 at three to six months and a non-significant g = 0.110 beyond six months; only 20% of effect sizes came from implementations longer than six months.
6. Measurement timing inflated effects. Outcomes measured immediately after the intervention yielded g = 0.460 against g = 0.155 at end of school year, and researcher-developed tests yielded a non-significant g = 0.483 against g = 0.200 on standardized tests.
7. Internal-validity reporting is thin. Only 8 of 18 studies reported differential attrition: five met What Works Clearinghouse standards without reservations, three with reservations, and the remaining ten could not be classified.

What the pooled effect does and does not establish

The review's value lies in what it separates. Previous meta-analyses of Intelligent Tutoring reported larger effects, but they mixed designs — quasi-experiments and observational studies alongside randomized trials — and pooled across countries, so their estimates carry both selection and attrition bias. Restricting to U.S. K-12 experimental evaluations produces a smaller number that is closer to what a district can expect when it adopts a system and compares it with ordinary classroom instruction. That comparison is the practically relevant one: one-to-one human tutoring is the benchmark these systems chase, but it is not the counterfactual a school actually faces. Against regular classrooms as the control, using the system as the main instructional method (g = 0.294) and for individual homework (g = 0.317) both worked, while using it as a separate activity (g = 0.235) or in after-school settings (g = 0.014) did not. Personalized Learning benefits appeared to require integration into the instructional core rather than an optional add-on.

The attrition finding is the quiet corrective. The authors hypothesized that studies with weaker internal validity would report larger effects, and found the opposite: effect sizes for studies with low or high attrition were similar in magnitude, and the only significant estimate (g = 0.251) came from studies that never reported attrition at all. That pattern does not prove the unreported studies were biased, but it does mean a reader cannot tell. For a field that invests heavily in randomized trials, a corpus in which ten of eighteen studies cannot be classified against a federal standard is a reporting failure independent of any effect size.

Which design features carry the effect

The moderator analysis is where the study earns its keep for learning design. Just-in-time hints — fired automatically when the system detects an error — were not significant, and the authors tie that null to the interruption account: automatic hints can cut across a student's own problem-solving and remove the productive struggle that makes practice instructive. On-demand examples and hints put timing under the learner's control, which is why the authors read their significance as support for student choice. That reading carries a caveat they state plainly: on-demand features can be gamed. Students with weaker self-regulation may request hints or answers rapidly without engaging, which is the well-documented Help-Seeking failure mode, so the design question becomes how to scaffold productive help-seeking rather than how to time the prompt. Neither Adaptive Learning nor item-level adaptivity showed up as the active ingredient; the visible design difference was what support students could summon and when.

Duration produced a shape worth planning around. Short trials can inflate effects through novelty and tight alignment between practice and the post-test, and very long implementations can deflate them through fatigue and disengagement. The significant band sat between three and six months, which is the range where students have learned to use the system but have not yet tired of it. The authors draw the obvious implication for scale: most of what is known comes from implementations short enough that the system is still novel, and almost nothing comes from a full year of ordinary use. Their recommendation is for longitudinal study of engagement, teacher orchestration and curriculum alignment, not another short efficacy trial.

What this means for practice

• Instructors. Prefer systems that let students pull worked examples and hints on demand over systems that interrupt with automatic prompts, and teach the help-seeking norm explicitly — request support when stuck, then return to independent work.
• Administrators. Budget for at least a semester of use and plan to measure outcomes at end of year, not at the end of the unit: immediate post-tests overstate effects by roughly a factor of three in this corpus (g = 0.460 vs 0.155).
• District and rural leaders. Expect smaller returns where devices and connectivity are thin (g = 0.146 in studies including rural schools); treat infrastructure as part of the intervention rather than as a precondition already met.
• Researchers. Report differential attrition and sample characteristics even when the design is cluster-randomized — ten of eighteen studies left reviewers unable to classify them, which caps what any meta-analysis can conclude.

Limitations

• Only 18 studies and 11 systems were eligible; several systems appear in a single study, which blocks between-system comparison and meaningful subgroup analysis.
• Grade coverage was uneven: just two studies targeted high school, so the estimates describe elementary and middle school far better than secondary.
• Gender, ethnicity and class size were missing for more than half the studies and had to be dropped as moderators, leaving the equity picture incomplete.
• The search found no eligible quasi-experimental designs, so design type could not be tested as a moderator even though rigorous quasi-experiments would have broadened the evidence base.
• Support types were unequally distributed across systems (hints are common in mathematics systems, rare in reading and writing), which limits power to detect moderation by any single feature.
• The authors' own MetaForest procedure cannot establish causal moderation; the moderator ranking is an association within a small, unbalanced set of studies.

Citation

Leite, W. L., Zhang, H., Rana, S., Hao, Y., Hatch, A. D., Kong, L., & Kuang, H. (2025). Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S.. arXiv:2511.04997.