One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment

Synthesis: Engagement, not model capability, is the binding constraint on AI tutoring. In a two-year cluster randomized trial across 18 Tennessee middle schools, randomly assigned students used Khan Academy with its AI tutor Khanmigo — configured to coach rather than give answers (a scaffolded guided-help design) — during existing daily remedial math sessions. Assignment raised math achievement by 1.3 national percentile ranks per term (~0.06–0.08 SD per school year; an implied ~0.14 SD for a full year of active participation), gains resembling those of Khan Academy practice without AI assistance. The reason: 96% of students tried Khanmigo at least once, but the median student messaged it on only about a third of practice days and in only 17% of the exercise sessions in which they made a mistake — and most messages were bare answers or clicks on suggested prompts, with just ~14.5% containing a genuine mathematical question or reasoning step. "Making AI effective for learning appears to be as much a behavioral challenge as a technological one": realized Help-Seeking, not access, is what limits LLM tutoring in routine school settings.

Key Findings

• Modest achievement gains at very low cost. Assignment raised math achievement ~0.06–0.08 SD per school year (1.3 national percentile ranks per term), delivered inside remedial blocks the district already staffed at a marginal license cost of roughly $15 per student per year — a cost structure that, unlike human tutoring, scales without adding skilled labor.
• The effect is primarily the structured practice, not the tutor. Gains resembled those from Khan Academy practice without AI assistance; estimates are best interpreted as the effect of structured, individually targeted practice, with the tutor's contribution bounded by its low realized use. High-dosage human tutoring still delivers far more per student (0.2–0.4 SD) — software does not close that gap.
• The AI tutor was barely engaged. 96% of students tried Khanmigo at least once, but the median student messaged it on only ~14% of exercise sessions and ~1/3 of practice days, and roughly one message in seven contained a genuine math question or reasoning step. The binding constraint was engagement, not what the tutor could do.
• Struggling students are least likely to seek help unprompted. Consistent with the economics-of-education literature, interventions that rely on student initiative reach fewest of the students who would benefit most — especially when the action is incremental (considering one mistake on one problem). The limited dialogue reflects a lack of interest in the marginal act, not a defect of the AI tutor.
• Design responds to the engagement margin. Khan Academy redesigned Khanmigo in 2026 to activate automatically during practice, citing lower student-initiated use than anticipated; evidence from other platforms indicates known human support raises engagement with AI tutoring where availability alone does not. Returns to AI-tutoring investment depend on the engagement margin as much as on capability.

What this means for practice

• Instructors. Treat the AI tutor as an add-on to structured, teacher-directed practice rather than a substitute for it: the gain came largely from mastery-based practice placed from MAP scores, since the median student messaged Khanmigo in only 17% of the exercise sessions where they made a mistake. Place the tutor inside a block you already supervise, and prompt help-seeking at the moment of a mistake rather than assuming students will initiate it.
• Instructors. Expect an idle tutor unless it is made structurally salient. Students messaged Khanmigo on only about a third of practice days when the tool was optional; the 2026 redesign that auto-activates it during practice, plus evidence that known human support raises engagement where availability alone does not, points to lowering the marginal cost of starting a tutoring exchange rather than improving its responses.
• Administrators. Budget from realized use, not capability. At roughly $15 per student per year the program delivers individualized-feedback time at a cost structure human tutoring cannot match — but the effect is bounded by actual dosage, so fund the engagement work (scheduling, teacher prompting, progress monitoring) alongside the license.
• Administrators. Set expectations against the right comparator. High-dosage human tutoring still delivers 0.2–0.4 SD, and most control sessions already used some computer-assisted learning (Waggle, IXL, Zearn, DeltaMath), so position the tool as a supplement to the district's existing remedial mix rather than a replacement for skilled adult attention.
• Researchers. Instrument message-level dialogue, not logins: access was near-universal (96% tried Khanmigo at least once) while only 14.5% of messages contained a genuine mathematical question or reasoning step, a gap that platform access metrics alone conceal.

Limitations

• Single district and single program: the trial ran in 18 Tennessee middle schools with 53 grade-within-school clusters, so estimates reflect one district's remedial schedule, staffing, and student population.
• The treatment bundled mastery-based practice structure with Khanmigo availability, and 14 of the 18 control schools used some computer-assisted learning (Waggle, IXL, Zearn, DeltaMath), so the contrast cannot isolate the AI tutor from the practice redesign or from business-as-usual software.
• Message-level chat records matched to exercise logs are available only for Year-2 treated students, so the engagement analysis rests on the single year of strongest implementation.
• The design has limited power on the cross-grade spillover margin and finds no detectable spillover signal, so contamination between treated and control grades cannot be ruled out where a teacher led RTI mathematics in both.

Citation

Oreopoulos, P., & Low, N. (2026). One click away: AI tutoring with Khanmigo in a two-year school experiment (NBER Working Paper No. 35620). National Bureau of Economic Research.