CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming

Deploying LLM tutors in K-12 raises concerns around privacy, cost, and reliance on proprietary models, motivating small language models (SLMs) as an alternative. The authors introduce CSTutorBench, a benchmark evaluating language models as CS tutors in VEX VR, a block-based robotics environment. It comprises 17 scenario-based questions scored against a pedagogical rubric grounded in tutoring and feedback research, using a human-in-the-loop LLM-as-judge pipeline. Across 11 models (4B–120B parameters), models handled surface-level criteria (vocabulary, tone) well but struggled with deeper pedagogical behaviors — especially avoiding answer leakage and engaging with student debugging histories. Model family and instruction-tuning predicted tutoring quality better than parameter count; a targeted prompt revision improved scores for 10 of 11 models.

K-12 privacy-aware tutoring: Motivates on-device SLMs for K 12 to address Privacy and cost vs proprietary LLMs.

Pedagogically grounded benchmark: 17 scenarios scored on a rubric tied to Intelligent Tutoring and Feedback Loop research, in block-based programming.

Depth gap: Models ace surface tone but leak answers and ignore debugging history — aligning with AI Tutor Behavioral Evaluation concerns.

Family over size: Instruction-tuning and model family beat raw parameter count, refining CS Education tutor selection.

Prompt-engineering leverage: A rubric-grounded prompt revision lifted 10/11 models, echoing Scaffolding practice.

Connected Concepts

Connected Articles

AI Tutor Behavioral Evaluation

Citation

Lane, H. C., & Kageler, B. (2026). CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming. arXiv:2607.05571.