H. Chad Lane & Bryson Kageler (2026) — University of Arizona / University of Illinois. arXiv.
Deploying LLM tutors in K-12 raises concerns around privacy, cost, and reliance on proprietary models, motivating small language models (SLMs) as an alternative. The authors introduce CSTutorBench, a benchmark evaluating language models as CS tutors in VEX VR, a block-based robotics environment. It comprises 17 scenario-based questions scored against a pedagogical rubric grounded in tutoring and feedback research, using a human-in-the-loop LLM-as-judge pipeline. Across 11 models (4B–120B parameters), models handled surface-level criteria (vocabulary, tone) well but struggled with deeper pedagogical behaviors — especially avoiding answer leakage and engaging with student debugging histories. Model family and instruction-tuning predicted tutoring quality better than parameter count; a targeted prompt revision improved scores for 10 of 11 models.
Key Contributions
- K-12 privacy-aware tutoring: Motivates on-device SLMs for k-12 to address privacy and cost vs proprietary LLMs.
- Pedagogically grounded benchmark: 17 scenarios scored on a rubric tied to intelligent-tutoring and feedback-loop research, in block-based programming.
- Depth gap: Models ace surface tone but leak answers and ignore debugging history — aligning with ai-tutor-behavioral-evaluation concerns.
- Family over size: Instruction-tuning and model family beat raw parameter count, refining cs-education tutor selection.
- Prompt-engineering leverage: A rubric-grounded prompt revision lifted 10/11 models, echoing scaffolding practice.
Related Pages
- k-12 — Block-based programming tutoring in K-12
- privacy — On-device SLMs vs cloud proprietary tutors
- intelligent-tutoring — Pedagogical rubric for tutor behavior
- cs-education — Computing and robotics education
- ai-tutor-behavioral-evaluation — Deeper pedagogical behaviors
- feedback-loop — Tutor feedback quality