Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Summary

Yasir et al. (2026) Benchmark seven LLM Feedback agents as step-level tutors in propositional logic, using Knowledge Graph-derived ground truth across 10,836 solution–feedback pairs and three feedback conditions. Their core finding: models achieve near-ceiling performance on optimal steps but systematically over-reject valid-but-suboptimal reasoning and over-validate incorrect solutions — precisely where adaptive tutoring matters most. These failures persist across models and solution contexts (suggesting architectural rather than informational limits), and accurate diagnosis does not reliably produce pedagogically actionable Feedback. The authors conclude LLMs are better suited to hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended Scaffolding and dialogue.

Key Findings

• Near-ceiling on optimal steps, failures elsewhere: models classified optimal solutions well (F1 94–99%) but struggled on valid-alternative solutions (F1 0–76%) and incorrect solutions (F1 4–55%).
• Over-rejection and over-validation (OR/OV): Gemini/DeepSeek over-validated (69–71%), treating most solutions as valid alternatives; LLaMA 3 over-rejected (91%), rejecting nearly all non-optimal solutions; GPT-4.1/GPT-o3 showed moderate rates of both.
• Model-level, not information-level, failures: model selection explained most variance (η² > 0.95); feedback conditions, step complexity, distance-to-goal, and inference rule had negligible effects. Failures persisted regardless of solution context.
• Accurate diagnosis ≠ pedagogically useful feedback: even correct classifications often produced encouraging but pedagogically misleading feedback — praising solutions without identifying errors or giving actionable guidance, especially for incorrect and valid-alternative solutions.
• Hybrid architecture indicated: LLMs suit open-ended Scaffolding and dialogue; KG-grounded models should handle diagnostic classification.

Implications

• Current LLM tutors cannot resolve the "assistance dilemma" without ITS-grounded diagnostic mechanisms.
• Effective LLM tutoring in structured reasoning domains requires hybrid architectures delegating diagnostic classification to KG-grounded models while using LLMs for Scaffolding and dialogue.
• Evaluation must move beyond binary correctness to three-way (optimal / valid-alternative / incorrect) classification to reveal these failure modes.

Connected Concepts

• Intelligent Tutoring
• Student Modeling
• Knowledge Graph
• Knowledge Tracing
• AI Feedback Quality
• Feedback
• AI Ed Evaluation
• Educational Measurement
• Scaffolding
• Socratic Method
• LLM

Connected Articles

• From Answer Generators To Reasoning Facilitators AI Tutors — From answer generators to reasoning facilitators
• LLM Tutoring Feedback Diagnosis Gap — LLM tutoring feedback diagnosis gap
• Zerkouk Comprehensive Review ITS 2025 — AI tutor effectiveness review
• Deeptutor — DeepTutor: agentic personalized tutoring

Citation

Yasir, T., Li, W., Gilson, S., Tithi, S. D., Tian, X., & Barnes, T. (2026). Confirming correct, missing the rest: LLM tutoring agents struggle where feedback matters most.