Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop

Synthesis: Adak, Hazra, Mukherjee, Bose & Banerjee (2026) reframe the LLM tutoring literature as a single process — a tutoring loop that runs from learner-state inference (L1) to pedagogical action (L2), wrapped by tutor training, evaluation, and student Simulation. They code 88 studies and find that the parts are common but the joints are not: tutoring components appear widely, yet only 9 of 88 studies (10.2%) connect a learner state, a learner-conditioned intervention, an explicit policy, and a downstream outcome. Evidence thins sharply as claims strengthen, and human studies return a conditional positive answer — LLM tutors can improve learning, but benefits depend on pedagogical structure, engagement, and how learning is measured. Three fast-moving threads get particular scrutiny: dialogue-native learner modeling, reinforcement-learning policies that must assign credit across turns, and simulated students that serve as the environment for both training and evaluation. The authors' thesis is that progress depends on validating the interfaces between stages rather than adding more components.

Key Findings

1. Across the coded corpus of 88 studies, pedagogical evaluation (77/88, 87.5%) and an explicit pedagogical policy (62/88, 70.5%) are widespread, whereas learner-state modeling (20/88, 22.7%) and tutoring-specific optimization (15/88, 17.0%) are substantially less common.
2. Only 9 of 88 studies (10.2%) jointly contain learner-state estimation, learner-conditioned intervention, an explicit pedagogical policy, and a downstream outcome — the corpus's "core adaptive path."
3. The bottleneck is not simply ignored learner states: 14 of the 20 studies with an explicit learner state (70.0%) use it to condition a later action, so the sharper gap is connecting the stages.
4. Evidence thins steeply with claim strength: 76 of 88 studies (86.4%) reach at least pedagogical-quality evidence, 27 (30.7%) reach downstream task or interaction evidence, 10 (11.4%) measure human learning, and one reaches transfer or delayed retention.
5. Of the 34 studies reporting a downstream learner or task outcome, 10 (29.4%) reach direct human-learning evidence and only one (2.9%) meets the strict transfer or delayed-retention criterion.
6. Human studies support a conditional yes: LLM-based tutoring can improve learning when assistance is pedagogically structured or adapted, but a system that increased engagement produced no significant short-term learning advantage over reading.
7. Student simulation, used for repeatable and counterfactual rollouts, carries five recurring validity hazards — fake confusion, competence or answer leakage, profile leakage, simulator–judge circularity, and mismatched learning dynamics.

Reading the loop as a design checklist

For an instructor or designer choosing a tutor, the loop is less a taxonomy than a checklist of questions about how a system is wired. A tutor may infer a learner profile and never use it, or run an explicit pedagogical policy with no learner state at all; a few connect the stages, such as LongTutor, which separates acquisition of historical evidence, diagnosis of the current knowledge state, and adaptive teaching. The authors' load-bearing point is that implementation is not validation: a learner state can be accurate but unused, a pedagogically plausible action can fail to teach, and a realistic-sounding simulator can react unrealistically to an intervention. Each joint therefore needs its own evidence — state to action, intervention to behavior, evaluator to independent human judgment. Read this way, the useful question about a new tutor is not which modules it has but which of its transitions were actually tested.

What the evidence ladder licenses

The ladder ranks the strongest learner-related claim a study can carry: E1 component or response evidence, E2 pedagogical quality, E3 downstream task or interaction behavior, E4 measured learning gains with real learners, and E5 transfer or delayed retention. That ordering changes how evaluation numbers should be read. A model that scores well on pedagogical quality has cleared E2, not shown learning; solving and teaching do not move together on a Benchmark such as MathTutorBench, which is why scoring shifted from correctness to constructs such as awareness of student understanding and instructional support. The denominators also matter: corpus-level percentages describe the coded corpus rather than population-level prevalence, because learner-modeling, evaluator, and simulator papers do not necessarily claim to demonstrate learning. And the ladder measures claim strength, not study quality — a careful E2 study can be better executed than a weak E4 one.

Simulated students can flatter the tutor

Student simulation is the environment most training and evaluation pipelines run on, buying cheap, repeatable rollouts — and the risk that a tutor optimized against one simulator learns simulator-specific shortcuts. More conditioning is not more fidelity: explicit learner attributes, prior responses, and ability-grounded response probabilities each constrain a different part of the simulation, and only comparison with real learners shows what mismatch remains. Teachers still flag unrealistic attentiveness, language, emotion, and consistency, and real-versus-simulated studies find residual gaps in dialogue acts, errors, and learning dynamics. The five hazards above all flatter the tutor, either by making the learner easier to teach or by rewarding what the simulator happens to favor. The practical rule the authors draw is that a simulator good enough for ranking task difficulty may be useless for evaluating Socratic tutoring, so papers should state what was validated, against what, and for which use.

Deployment: access is not use

The loop reframes deployment as an evaluation concern rather than an afterthought. Long-term access to Khanmigo did not automatically produce substantive tutoring interaction, which separates availability from effective tutoring; a five-month high-school deployment paired an LLM tutor with reinforcement learning for practice sequencing and, according to the authors, adaptive sequencing beat a fixed sequence on unassisted final-exam performance. Immediate success and durable learning can also diverge: delayed assessment shows the two need not move together, and a conversational system that raised engagement produced no significant short-term learning advantage over reading. Taken together these strands support the paper's conditional answer to its title question — LLM tutors can teach, but fluent dialogue and high pedagogical-quality scores are insufficient by themselves. Benefits depend on learner evidence, pedagogical action, engagement, and whether evaluation measures performance, learning, retention, or transfer.

What this means for practice

• Instructors. Ask which stage a tutor actually connects before trusting it: a system can hold learner modeling, policy, and evaluation separately without ever joining them, so check whether the tool shows it used what it inferred about the student.
• Instructors. Treat pedagogical-quality scores as a floor, not proof of learning: in the coded corpus 76 of 88 studies (86.4%) reach pedagogical-quality evidence, but only 10 (11.4%) measure human learning, so look for learning gains measured with real learners.
• Researchers. Validate interfaces, not just components: test state-to-action sensitivity, evaluator calibration against independent human judgments, and simulator fidelity against real-learner behavior, because a learner state can be accurate yet unused.
• Administrators. Judge procurement on use rather than access: long-term access to a capable tutor did not automatically produce substantive tutoring interaction, so track engagement and, where possible, delayed outcomes rather than adoption alone.

Limitations

• The literature moves faster than the search: work published after the cutoff of August 31, 2026 is not represented.
• Scope is restricted to conversational LLM-based tutoring in which language is a primary interaction mechanism, so classical intelligent tutoring systems and non-conversational applications are included only as background.
• The tutoring-loop coding reduces heterogeneous systems to a common set of operational categories, so borderline cases remain possible, especially for learner-conditioned action and the distinction between adaptive components and genuinely closed-loop adaptation.
• The evidence ladder measures the strength of the learner-related claim a study supports, not the overall methodological quality of that study.
• The human-learning literature is heterogeneous in domain, learner population, intervention duration, comparison condition, and Assessment design, so the authors deliberately do not pool reported effects into a meta-analytic estimate.
• The corpus is concentrated in mathematics and programming, where correctness and learning outcomes are comparatively easy to operationalize, so the conclusions may not transfer to open-ended, social, creative, or multilingual domains.

Citation

Adak, S., Hazra, R., Mukherjee, K., Bose, D., & Banerjee, S. (2026). Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop. arXiv preprint arXiv:2610.15692.