Schleifer, Ariely & Klebanov (2026) investigate a critical gap in Automated Grading: how scoring quality degrades for mid-range student responses. Most ASAS evaluations focus on clearly correct or incorrect answers, but real classrooms are dominated by partially correct responses where scoring is most challenging.

Automated Short Answer Scoring: Mid-Range Quality Degradation

Core Contribution

Schleifer, Ariely & Klebanov (2026) investigate a critical gap in Automated Grading: how scoring quality degrades for mid-range student responses. Most ASAS evaluations focus on clearly correct or incorrect answers, but real classrooms are dominated by partially correct responses where scoring is most challenging.

Key Findings

The paper reveals that automated short answer scoring (ASAS) systems show significant quality degradation in the mid-range — exactly where teacher judgment is most needed. This connects directly to Automatic Short Answer Grading research on confidence-aware LLM grading with epistemic uncertainty quantification. The finding that task-specific adaptation can mitigate this degradation provides a practical path forward.

Significance for AIED

This work fills a gap in the AI Tutor Behavioral Evaluation landscape: Niousha et al.'s 10K-student analysis identified missing evaluation axes for AI tutoring, and mid-range scoring reliability is one such axis. The quality-conditioned agreement approach offers a more nuanced alternative to simple accuracy metrics used in benchmark evaluations.

The findings also matter for Formative Assessment systems — if ASAS works well only at extremes, it may reinforce binary thinking rather than supporting the nuanced feedback that Sequenced AI Feedback Learning research shows is critical for learning. The connection to Human In The Loop AI is clear: mid-range responses may be where human teacher judgment remains essential.

Connections to Wiki

Connected Concepts

Connected Articles

Citation

Klebanov, A.A.V.G.S.M.A.B.B., Scoring:, Q.A.I.A.S.A., Adaptation, M.D.A.T.I.O.T., Klebanov2, A.V.G.S.M.A.B.B., Alexandron1, A.S.G., & Princeton, E. (2026). Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation. par- require ample training data (Gurin Schleifer et al