EduGuard: A Safe RAG-Based LLM Tutor for Programming Education

Synthesis: EduGuard is a retrieval-augmented generation (RAG) tutoring framework that directly confronts the safety and pedagogical failures of unrestricted Large Language Models (LLMs) tutors in introductory programming. Unrestricted tutors hallucinate, contradict course policy, reveal complete solutions, and foster passive dependence; EduGuard counters these with query understanding, instructor-approved course retrieval, pedagogical-strategy selection, rubric-aware generation, claim-level verification, and explicit overreliance control. The design maps onto the SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems taxonomy and directly targets Hallucination Risk and Over-Reliance in deployed tutors.

To make evaluation provenance explicit the authors build BILearn-CS, a 600-query instructor-authored, TA-validated Benchmark spanning concept questions, debugging, Misconceptions about AI, assignment support, code-mixed Bangla-English queries, and adversarial direct-answer prompts, then validate on a public CS50-style forum set and a small n=10 pre/post-test pilot. This contributes a rare benchmark for safe tutoring and extends the Programming Intelligent Tutoring Systems and A Classroom Study of LLM-Generated Feedback Intervention in Introductory Programming literature with a verifier-separated architecture (DeBERTa-v3-large-MNLI as an architecturally separate claim verifier) that improves on The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance baselines and GPT-4o-mini and Llama Socratic tutors. The work is also a concrete instance of Training Pedagogical LLMs for Tutoring where safety and pedagogy are baked into the generation pipeline rather than retrofitted.

What this means for practice

• Instructors. Constrain tutor retrieval to instructor-approved course materials: EduGuard's restricted retrieval targets the incomplete-retrieval and hallucination failures that dominate its 80-case failure analysis on BILearn-CS.
• Instructors. Keep claim verification architecturally separate from generation, as the DeBERTa-v3-large-MNLI verifier is, but do not assume independence: the authors note both models are trained on broad web data and can fail in correlated ways.
• Instructors. Treat code semantics as the residual weak point - the NLI verifier can flag unsupported natural-language claims but cannot determine whether a code trace or edge case is logically correct without executing it.
• Instructors. Deploy with oversight rather than autonomously: student-facing use should include instructor control, given that the evidence base is a 600-query benchmark, a 150-query course-forum set and a 10-student pilot.
• Instructors. Relabel the leakage policy for your own course, since what counts as a prohibited full solution is partly contextual: rubric entries must be tagged as explanation allowed, hint only, example allowed or full solution prohibited.

Limitations

• The classroom evidence comes from a controlled pilot with 10 undergraduates in a counterbalanced within-subject design; the authors state it is underpowered and supports feasibility and short-term learning indicators only, not long-term retention or course-grade improvement.
• No grades were affected and participants were told not to submit generated content as coursework, so the pilot measures immediate post-test accuracy (68.4% to 81.2%) rather than authentic classroom performance.
• BILearn-CS is a 600-query instructor-authored, TA-validated benchmark, and the external set is 150 public CS50-style forum questions treated as a generalization test rather than a classroom deployment, with incomplete code context and wider confidence intervals.
• The leakage metric carries a tautology risk because policy-based leakage depends on instructor assistance labels; the authors reduce but cannot remove it with rule-based checks on observable output features such as complete runnable code and final-answer disclosure.

Citation

Hossain, S. M. A., Shayoni, R. K., Mridha, M. F., & Shin, J. (2026). EduGuard: A Safe RAG-Based LLM Tutor for Programming Education.