Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment

Synthesis: Making AI tutoring productive depends on the structure that surrounds it — not just the model. In a randomized field experiment with 6,000+ middle-school students using NUMI, a research-based computer-assisted learning platform, students assigned to AI support progressed more slowly and attempted fewer questions, but answered more accurately and — the clearest mechanism — improved their next-attempt correctness after mistakes, needing fewer attempts to return to a correct answer while spending more time on each structured question. This is a "productive slowdown": AI that coaches rather than gives answers takes time, but turns mistakes into learning. Notably, a mastery rule (three-correct-in-a-row) sharply increased platform-defined success without by itself improving delayed learning — a short-run streak is not the same as durable understanding. The strongest delayed-test evidence emerged when AI was embedded in the mastery workflow, concentrated on practiced material.

Key Findings

• AI slows progress but improves conditional accuracy. Students randomized to AI support completed fewer questions and progressed more slowly through the assignment, but were more accurate conditional on reaching an attempt — consistent with an AI tutor that scaffolds rather than simply supplies answers.
• The clearest mechanism is post-mistake recovery. AI increased next-attempt correctness after errors and reduced the attempts needed to return to a correct answer, while increasing clock time per question under structured support — the sense in which AI created a productive slowdown rather than shallow answer-grabbing that turns effort avoidance into learning.
• The "mastery puzzle." Requiring three correct answers in a row substantially raised platform-defined mastery and practice, yet did not by itself produce detectable delayed-test gains one week later. Reaching a short-run streak is an imperfect proxy for durable understanding — some students clear the threshold through repeated exposure, luck, or guessing rather than comprehension, echoing concerns about engagement quality over quantity.
• AI's value is conditional on structure. The most encouraging delayed-test evidence appeared when AI was embedded in the mastery workflow (Mastery × AI practiced-delayed coefficient 0.085), with marginally significant gains concentrated on practiced Exercise 1 material. AI access alone added little; AI paired with a structured workflow that makes mistakes consequential is where the value lives.
• Practical implication: designing AI tutoring productively means embedding the tutor in the practice environment, surfacing help at moments of need, and pairing it with progression rules that make errors matter — turning effort avoidance into productive scaffolded struggle.

What this means for practice

• Designers. Embed the tutor inside the practice environment rather than offering a passive chat window: mastery students averaged 1.99 "help me get started" uses, 2.28 post-mistake walkthroughs, and 3.23 step explanations, and the measured effect came from those structured touchpoints.
• Instructors. Pair AI support with a progression rule that makes errors consequential — the strongest delayed-test evidence came from AI embedded in the mastery workflow, while AI access alone added little.
• Designers. Report platform-defined success separately from durable learning: mastery raised three-correct-in-a-row attainment by roughly 28.7 percentage points without improving delayed-test performance, so streak metrics should not stand in for learning on dashboards.
• Instructors. Protect the time a "productive slowdown" needs: AI students completed fewer questions (1.04 fewer in the non-mastery condition) and spent more clock time per supported question, but answered more accurately and returned to a correct answer in fewer attempts.

Limitations

• The study ran in one district — Hamilton County Schools (Chattanooga, Tennessee), grades 6–8 across 20 schools and just under 100 teachers — so results may not transfer to other student populations or settings.
• The intervention was short: one roughly 50-minute NUMI session in March 2026 followed by a 15–20-minute delayed assessment about one week later; the authors state this does not support claims of transformational learning gains.
• The delayed-test estimates are imprecise — the Mastery × AI coefficient on practiced Exercise 1 is 0.085 and only marginally significant, and the combined practiced-minus-unpracticed contrast is too imprecise to support a large or uniform effect.
• The experiment cannot separately identify why mastery raised streak attainment (understanding versus repeated exposure or guessing), and later exercise outcomes are contaminated by differential progression and remaining class time.

Citation

Oreopoulos, P., Liut, M., Sungu, A., & Low, N. (2026). Making AI tutoring productive: Evidence from a mastery-based math practice experiment (NBER Working Paper No. 35621). National Bureau of Economic Research.