Mihaylova, T., Logacheva, E., Hellas, A., Fan, J., Castro, F., Akram, B., Norouzi, N., Brusilovsky, P., & Leinonen, J. (2026) — Aalto University, University of Pittsburgh, UC San Diego. arXiv preprint.
Key Finding
LLM-generated feedback produces faster time-to-solution than compiler-only baseline; counterintuitively, less guided feedback showed stronger effects than more guided variants.Synthesis
This study provides empirical evidence on how LLM-generated feedback structure affects programming students' problem-solving. In an online programming course, three LLM feedback types (varying guidance levels) were compared against a compiler-error-only baseline. All LLM feedback conditions led to faster time-to-solution, but surprisingly, less guided feedback showed slightly stronger effects than more directive variants. This aligns with socratic-questioning principles and the desirable-difficulties framework, suggesting that too much hand-holding may short-circuit productive struggle. The finding also resonates with the codify-socratic-tutoring-programming approach using discovery-based learning over direct answer generation. Combined with ai-tutor-behavioral-evaluation, which showed that behavioral engagement metrics matter more than feedback quality ratings alone, this paper strengthens the case for designing LLM feedback that encourages student action rather than passive consumption. Future work should explore adaptive feedback that calibrates guidance based on knowledge-tracing-irt student models and individual programming-its experience levels.Related Pages
- mindcopilot-llm-co-writing — Behavior-centered evaluation framework applicable beyond writing to programming feedback
- cyberscholar-genai-writing-feedback — Generative AI Feedback, English Writing and Teacher Rubrics: A Multiple-Case Study of CyberScholar
- socraticode-k12-programming-tutor — Towards SocratiCode: Designing a Generative AI-Based Programming Tutor for K-12 Students through a 4-Week Participatory Design Study
- eduguard-safe-rag-llm-tutor — Outperforms rubric-prompting and same-model RAG baselines via claim-level verification.
- llm-programming-support-governance-cs-education — Exploring the Design Space of LLM-Based Programming Support