Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education

This paper presents a rigorous empirical comparison between LLM-based and semantic similarity methods for automated assessment of student self-explanations in programming education. The task is framed as binary classification — determining whether a student's explanation of a worked-example step is correct or incorrect.

Worked examples — step-by-step problem solutions — are a well-established Scaffolding technique, and their effectiveness increases when students are prompted to self-explain each step. However, manually assessing these self-explanations doesn't scale. The prevailing approach has been to compare student responses to reference explanations using semantic similarity metrics, but recent advances in large language models raise the question of whether LLM-based scoring now outperforms these traditional methods.

The authors address a critical gap: high-quality, domain-specific datasets with balanced class distributions for automated scoring tasks. Their contribution is both methodological (a rigorous comparison framework) and empirical (which approach works better, and under what conditions).

Connection to Wiki

This work extends the Automated Grading landscape by addressing a specific gap: assessment of open-ended self-explanations rather than final answers or code submissions. It complements research on Automatic Short Answer Grading and Structured LLM Feedback Programming by focusing on the formative, metacognitive dimension of student learning rather than summative evaluation.

Connected Concepts

Connected Articles

Citation

Lekshmi-Narayanan, A.-B., Hassany, M., & Brusilovsky, P. (2026). Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education. arXiv:2605.21614.