AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice

Synthesis: Thomas et al. (2026) present an AI-driven system that assesses human tutor training performance and links it to real-life tutoring quality, bridging the gap between training metrics and classroom practice. Unlike platforms that train tutors without evaluating them, this system assesses both open responses during training and authentic real-life tutoring, connecting tutor training to formative Feedback and AI-based assessment.

AI-driven assessment of human tutor training performance correlates with real-life tutoring quality, bridging the gap between training metrics and classroom practice. Numerous tutor training platforms exist, but few provide AI-driven training and evaluation for human tutors based on real-life performance. This system assesses both open responses during training and authentic real-life tutoring, unlike platforms that only train. The work demonstrates how AI-driven assessment can validate that the skills tutors practice in training transfer to genuine teaching, supporting tutor development and professional learning.

What this means for practice

• Instructors. Score tutors on the full battery of response formats rather than the cheapest ones: pooled lesson performance across 433 attempts was the best predictor of real-life tutoring quality (AIC = 1129.7), ahead of open-response-only (AIC = 1130.0) and multiple-choice-only models (AIC = 1150.0).
• Instructors. Set expectations that tutoring skill accrues gradually — tutors gained 7.4% within lessons (p < .001) and moved real-life quality from 65.5% to 68.1% (p = .003), with only a small linear trend over time (β = 0.01, p = .022) and no jump at the point of training.
• Instructors. Treat AI transcript scoring as a low-stakes formative signal only: with inter-rater reliability ranging widely across moves, the authors insist evaluation prompts need continuous auditing before such scores carry weight.
• Instructors. Add scenario practice for the move that did not transfer — tutors were weakest at recognizing when a reaction to a student error was warranted, an ability the scenario-based lessons never explicitly trained.

Limitations

• Lesson and transcript scores frequently hit ceiling effects (pretest means from 0.70 to 0.98, with only 66–76 tutors per lesson across 433 pooled attempts), which the authors say constrained the evidence of transfer from lessons to practice.
• Tutors uploaded their own tutoring sessions, creating a selection bias toward more confident tutors who submitted more data.
• Gemini-2.5-pro's inter-rater reliability varied widely across tutor moves, so the LLM-based evaluation is described as not a perfect process.
• Growth was inferred from a design in which the authors found no significant immediate performance change at the point of intervention and no evidence that opportunities to apply skills explained the observed trend.

Citation

Danielle R. Thomas, Marie Cynthia Abijuru Kamikazi, Clara Brandt, Conrad Borchers, Kenneth R. Koedinger (2026). AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice.