Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact

Studies whether public LLM tutoring benchmarks distinguish learning-supportive behavior from mere answer production. Proposes a lightweight diagnostic based on the gap between solving-oriented and pedagogy-oriented benchmark performance. Using MathTutorBench, shows correlation between solving and pedagogy composites is only r=0.421 across 8 models, with several models shifting rank when evaluated on pedagogy. Benchmarks reward guiding questions, calibrated hints, and non-disclosive scaffolding. Recommends reporting solving and pedagogy scores separately.

Key Findings

Stronger task-solving ability does not necessarily imply stronger learning support: a response that looks helpful can be pedagogically harmful by giving away the answer.

Using public MathTutorBench leaderboard results across eight publicly reported models, the correlation between solving-oriented and pedagogy-oriented composite scores is only r = 0.421, and several models shift meaningfully in rank when evaluation moves from solving to pedagogy.

Analysis of the public TutorBench sample shows that agency-relevant behaviors are explicitly encoded in benchmark rubrics, especially in active-learning settings that reward guiding questions, calibrated hints, and non-disclosive scaffolding.

The proposed diagnostic is lightweight: it is based on the gap between solving-oriented and pedagogy-oriented benchmark performance, requiring no new data collection.

The authors argue that educational-impact evaluation should not treat task success as a sufficient proxy for learning support.

Study Design & Method

The diagnostic exploits the fact that public tutoring benchmarks (MathTutorBench, TutorBench) score models on multiple rubrics. By separating rubric items into solving-oriented and pedagogy-oriented composites, the authors compute a per-model gap that reveals whether a model's benchmark standing reflects teaching quality or merely answer production. The correlational analysis across eight models quantifies how partially aligned the two dimensions are, while the rubric analysis identifies which specific behaviors — guiding questions, calibrated hints, non-disclosive scaffolding — benchmarks already reward.

Implications for AI in Education

For the Benchmark community and for AI tutor deployment, the findings argue for reporting solving-oriented and pedagogy-oriented scores separately and for making disclosure-sensitive, student-agency-preserving criteria more explicit. A model that tops a solving leaderboard should not be assumed to be a good tutor; evaluation infrastructure must measure learning support directly. This connects to Scaffolding and to the design of AI Tutoring systems where the goal is not the fastest answer but durable student understanding.

Connected Concepts

AI Tutoring

Pedagogical LLM Training

Connected Articles

LLM Judged Helpfulness Pedagogy Signal — Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models

Cstutorbench Slm Tutors — CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming

Human AI Collaboration Prerequisite Functions — What do you mean by human-AI collaboration: Prerequisite functions and the affordances needed to achieve it

Persistent AI Agents Academic Research — Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study

Rethinking Scaffolding LLM Tutors — Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

Multi Agent LLM Social Learning — Beyond the AI Tutor: Social Learning with LLM Agents

Citation

Junyi Yao, Zihao Zheng, Baichuan Li (2026). Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact. arXiv:2606.16206. arXiv preprint.