The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

Key Finding

LLM-generated feedback produces faster time-to-solution than compiler-only baseline; counterintuitively, less guided feedback showed stronger effects than more guided variants.

Synthesis

This study provides empirical evidence on how LLM-generated feedback structure affects programming students' problem-solving. In an online programming course, three LLM feedback types (varying guidance levels) were compared against a compiler-error-only baseline. All LLM feedback conditions led to faster time-to-solution, but surprisingly, less guided feedback showed slightly stronger effects than more directive variants. This aligns with Socratic Method principles and the Desirable Difficulties framework, suggesting that too much hand-holding may short-circuit productive struggle. The finding also resonates with the Codify Socratic Tutoring Programming approach using discovery-based learning over direct answer generation. Combined with AI Tutor Behavioral Evaluation, which showed that behavioral engagement metrics matter more than feedback quality ratings alone, this paper strengthens the case for designing LLM feedback that encourages student action rather than passive consumption. Future work should explore adaptive feedback that calibrates guidance based on Knowledge Tracing IRT student models and individual Programming ITS experience levels.

Connected Concepts

Connected Articles

Citation

Mihaylova, T., Logacheva, E., Hellas, A., Fan, J., Castro, F., Akram, B., Norouzi, N., Brusilovsky, P., & Leinonen, J. (2026). The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance.