Confidence-Aware Automatic Short Answer Grading

Confidence-Aware ASAG — A hybrid confidence estimation framework for Automatic Short Answer Grading with LLMs that fuses model-based confidence signals (verbalized, latent, consistency-based) with dataset-derived aleatoric uncertainty via Random Forest + Platt scaling, enabling reliable selective prediction and principled human-in-the-loop review workflows.

Key Findings

1. Model-based confidence alone is insufficient. Three LLM confidence signals were systematically compared — verbalizing (prompting the model to self-report confidence), latent (softmax-normalized conditional log-likelihood of the predicted label token), and consistency-based (agreement fraction across 5 predictions at temperatures [0.2–1.0]) — and none individually provided reliable uncertainty estimates for ASAG.

2. Aleatoric uncertainty must be explicitly modeled. The key innovation is modeling dataset-derived (aleatoric) uncertainty via semantic heterogeneity: student responses are embedded with all-MiniLM-L6-v2, clustered with agglomerative hierarchical clustering (Ward linkage), and within-cluster entropy quantifies inherent ambiguity. This captures the fundamental reality that some student answers are genuinely ambiguous, independent of model quality.

3. Hybrid fusion outperforms single-source approaches. Combining model-based signals with aleatoric uncertainty via a Random Forest classifier (500 trees) followed by Platt scaling produced the best calibration, evaluated on both selective prediction (AUROC / AUARC) and reliability (ECE / MCE / Brier score). The hybrid framework acknowledges that both epistemic and aleatoric uncertainty contribute to grading unreliability.

4. Principled evaluation enables safe deployment. By evaluating with selective prediction metrics rather than raw accuracy, the framework enables a practical workflow: high-confidence predictions can be auto-graded, while low-confidence cases are routed to human review. This is a more realistic deployment model than assuming perfect automation.

Implications

This work addresses a critical gap in Automatic Short Answer Grading: the gap between knowing that LLMs can grade short answers and trusting them enough to deploy in high-stakes settings. The central insight — that model confidence signals are unreliable on their own — is consistent with broader findings in Confidence Aware AI Assessment and psychometrically-aware-ai, which argue that AI assessment systems must be calibrated against human judgment uncertainty, not just accuracy.

The explicit modeling of aleatoric uncertainty is particularly important for Automated Grading. In ASAG, some student responses are genuinely ambiguous even to expert human raters — this is not a model failure but a property of the task. By modeling this inherent ambiguity through semantic clustering, the framework prevents the system from overconfidently misclassifying responses that are fundamentally uncertain.

For Human In The Loop AI workflows in educational assessment, the selective prediction approach provides a principled mechanism for triage: rather than requiring educators to review everything (defeating the purpose of automation) or trusting the model blindly (risking harm), the confidence framework identifies exactly which responses need human attention. This has practical implications for Short Answer Scoring Quality Degradation research and the design of Hybrid E Assessment Semi Automated Grading systems.

The use of RAG and synthetic-data techniques for calibration could extend this framework beyond the SciEntsBank domain, enabling confidence-aware grading across diverse subject areas and languages.

Connected Concepts

AI Ed Evaluation

Automated Grading

Confidence Aware AI Assessment

Hybrid E Assessment Semi Automated Grading

psychometrically-aware-ai

RAG

Connected Articles

Automatic Short Answer Grading — Automatic Short Answer Grading with LLMs

Short Answer Scoring Quality Degradation — Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specifi...

Aaai2026 Prompting Literacy K12 — Learning to Use AI for Learning: Teaching Responsible Use of AI Chatbot to K-12 Students Through an AI Literacy Module

Agreement Not Quality LLM Coding Verification — Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not G...

AI Assessment Human Tutors — AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice

AI Assessment Scale Reform — A bit of chaos and madness": The AI Assessment Scale and the work of assessment reform

AI Assistance Discretionary Feedback — AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education

AI Scoring Language Bias Physics — AI-based scoring systematically underestimates conceptual understanding of linguistically weak students' explanations...

Aiawe Automated Writing Evaluation — AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models

Aicode Collaborative Feedback System — AICoFe: Implementation and Deployment of an AI-Based Collaborative Feedback System for Higher Education

Aissa Slides Analysis — AISSA: AI-based Student Slides Analysis Tool for Academic Presentations

Authentic Assessment — Authentic Assessment

Authentic Products Authenticated Processes 2026 — From authentic products to authenticated processes: authentic assessment in AI-rich higher education

Automated Formative Assessments A Level Sciences — The Effect of High-Frequency, Automatically-marked Formative Assessments on Student Outcomes in A-Level Sciences

Automated Grading Linux Bash Examinations Large Language Models — Automated Grading of Linux/Bash Examinations Using Large Language Models

Becerra Aicofe Feedback 2026 — AICoFE: AI-Powered Feedback System

Behaviorally Adaptive Visual Diversion Assessment 2026 — Behaviorally Adaptive Visual Diversion for Inclusive and Resilient Digital Assessment Delivery

Beyond Detection Authentic Assessment AI 2025 — Beyond Detection: redesigning authentic assessment in an AI-mediated world

Calibrating Trustworthiness LLM Education 2026 — Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education

Code Gen — CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation

Confidence Aware Student Drawing Assessment — Confidence-Aware Automated Assessment of Student-Drawn Scientific Models

Correct Answer Trap AI Tutor — Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning

Correct Answer Trap Misconceptions — The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions

Cotal Formative Assessment Scoring 2026 — CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback

Cross Dataset Bloom Question Classification — Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs

Citation

Cong, L., Hahn, S., Gombert, S., Camus, L., Drachsler, H., & Kroehne, U. (2026). Confidence Estimation in Automatic Short Answer Grading with LLMs. arXiv:2605.00200.