Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Synthesis: Yasir et al. (2026) Benchmark seven LLM tutoring agents on three-way, step-level diagnosis — classifying a student's next proof step as optimal, valid-alternative, or incorrect — across 10,836 LLM-simulated solution–feedback pairs grounded in a Knowledge Graph solution space for propositional logic. Models reached near-ceiling performance on clearly optimal steps but misfired exactly where adaptive tutoring matters: they over-rejected valid-but-suboptimal reasoning and over-validated incorrect solutions. The failures were driven by model selection rather than solution context, step complexity or proof position, and accurate diagnosis did not reliably yield pedagogically actionable feedback. The authors argue for hybrid architectures in which KG-grounded models own diagnostic labeling while LLMs support open-ended Scaffolding and dialogue.

Key Findings

1. Near-ceiling on optimal steps, collapse elsewhere. Every model classified optimal solutions almost perfectly (F1 94–99%) but struggled with valid-alternative (F1 0–76%) and incorrect (F1 4–55%) solutions.
2. Two pedagogically distinct failure modes. Over-rejection labels valid reasoning as incorrect and discourages exploration; over-validation labels incorrect solutions as valid and reinforces Misconceptions about AI. Models skewed hard to one extreme — LLaMA 3 over-rejected 91% of the time, Gemini and DeepSeek over-validated 69–71%.
3. Failures are model-level, not informational. Model selection explained nearly all variance (η² > 0.95, p < .001) while feedback condition, step complexity, distance to conclusion and inference rule had negligible effects (η² < 0.01): richer solution context did not improve diagnosis.
4. Diagnosis does not translate into actionable Feedback. Even correct classifications often produced encouraging but misleading guidance — praise without error identification — and the Judge role tended to anchor on and propagate the Peer's mistakes rather than correct them.
5. Hybrid architecture indicated. LLMs suit open-ended dialogue; a KG-grounded model should own diagnostic classification, with feedback conditioned on the classifier's output rather than the model's self-assessed reasoning validity.

Why diagnosis was tested this way

Effective tutoring has to separate optimal, valid-but-suboptimal and incorrect reasoning: treating valid reasoning as wrong discourages exploration, while accepting every valid alternative without guidance reinforces inefficient strategies — the assistance dilemma. Classic intelligent tutoring systems resolve it by diagnosing each step against an expert-authored solution graph, but those graphs are costly to build for new domains and offer little conversational flexibility. LLMs promise open-ended dialogue and cross-domain generalization, yet ungrounded models hallucinate and often reveal answers instead of Scaffolding reasoning.

The authors extract 516 unique proof states from a propositional logic tutor used in an undergraduate discrete mathematics course (32 problems, five difficulty levels) and encode each problem's full solution space as a knowledge graph over 15 inference rules, so every valid derivation is enumerable. Seven models act in two roles — student simulator and feedback agent — under three prompt conditions (Peer: answer only; Teacher: full derivational context; Judge: context plus the Peer's feedback).

What drove the misdiagnosis

Model choice dominated classification variance, and the extra information in the Teacher and Judge conditions did not help when student reasoning diverged from the expert path. Step complexity predicted misdiagnosis only for optimal solutions and explained negligible variance overall, so the failures reflect model-level diagnostic bias rather than problem difficulty. Over-rejection peaked when solutions were closest to the conclusion and concentrated on structurally complex rules such as Modus Tollens, Constructive Dilemma and De Morgan.

Human raters then scored the feedback itself lowest on error identification and actionability. Contrary to prior LLM-as-Judge findings, judge verification did not consistently improve quality, and Teacher and Judge agents tended to penalize valid-alternative reasoning by steering students back to the provided solution.

What follows for practice and evaluation

Evaluation of tutors needs three-way, ground-truth-grounded diagnosis to expose over-rejection and over-validation that binary correctness checks hide. Verification requires an independent diagnostic signal — a separate model or an explicit reasoning chain enforcing independent judgment — not sequential refinement of a flawed judgment. Until that exists, current LLM tutors risk pedagogically misleading feedback precisely where adaptive support matters most.

What this means for practice

• Instructors. Do not let an LLM tutor hold sole judgment over student reasoning: LLaMA 3 over-rejected 91% of valid-alternative steps while Gemini and DeepSeek over-validated 69–71% of incorrect ones, so reserve unassisted LLM feedback for low-stakes use.
• Instructors. Confirm optimal steps and explicitly acknowledge sound reasoning that diverges from the textbook path, so students are not steered back to the provided solution.
• Designers. Delegate step-level diagnosis to a KG-grounded classifier and use the LLM for open-ended dialogue and hints, conditioning feedback on the classifier's output instead of the model's self-assessed reasoning validity.
• Researchers. Evaluate tutors with three-way, ground-truth-grounded diagnosis (optimal / valid-alternative / incorrect), because binary correctness checks hide the over-rejection and over-validation failure modes.
• Researchers. Require an independent diagnostic signal — a separate model or an explicit reasoning chain — rather than sequential refinement, since the Judge role anchored on and propagated the Peer's mistakes and did not consistently improve feedback quality.

Limitations

• All 10,836 solution-feedback pairs were generated from LLM-simulated next-step solutions rather than authentic student reasoning; alignment between simulated and real student solution patterns is untested.
• The 516 proof states come from a single undergraduate discrete mathematics course at one institution, spanning five difficulty levels.
• The design captures single-step feedback only, with no multi-turn student–tutor interaction and no cumulative learning-gain measure; feedback agents were also zero-shot, after prompt refinement.
• Findings are confined to propositional logic, where the authors can exhaustively enumerate valid inference paths in a knowledge graph; domains without tractable solution-space enumeration may behave differently.

Citation

Yasir, T., Li, W., Gilson, S., Tithi, S. D., Tian, X., & Barnes, T. (2026). Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most.