Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models

Synthesis: Pecuchova, Benko, and Drlik (2025) systematically benchmark eleven state-of-the-art GenAI and sentence embedding models against two expert human graders for the automated grading of 1,885 open-ended student responses to 24 questions from a software engineering course in higher education. Using inter-rater agreement metrics (Fleiss' Kappa, Krippendorff's Alpha, QWK) and classification error analysis, they find that GPTo1 achieves almost-perfect agreement with human graders, followed closely by Claude3 and PaLM2, while reference-based sentence embedding models such as BERT, RoBERTa, and T5 struggle with the diverse, context-sensitive phrasing students use. The work advances generative AI-driven Assessment and Educational NLP by showing that context-sensitive models outperform similarity-based approaches for grading open-ended responses.

Key Findings

Comparative benchmarking. Eleven models — GPTo1, Claude3, PaLM2, Mistral7B, Llama2, Cohere, SBERT, USE, T5, RoBERTa, and BERT — were evaluated on 1,885 answers from 110 students across 24 open-ended questions, against a benchmark set by two human graders in almost-perfect agreement (QWK = 0.95), providing a rigorous empirical basis for model selection in Automated Assessment.

GPTo1 leads in alignment with human graders. GPTo1 was the only model to reach almost-perfect agreement under the strict Fleiss' Kappa (0.82), with low false-positive and false-negative rates (11 FP, 39 FN) across grade categories including failing grades, making it the strongest candidate for independent deployment in higher education.

Claude3 and PaLM2 perform strongly but less completely. Both achieved substantial agreement with humans (Fleiss' Kappa 0.75–0.80) and near-perfect Krippendorff's Alpha, excelling at higher grades ('A', 'B', 'E') yet showing slightly reduced recall for identifying failing ('Fx') grades.

Context-sensitivity beats reference matching. Sentence embedding and reference-based models (BERT, RoBERTa, T5, USE) relied on cosine similarity to reference answers and systematically misclassified valid responses phrased differently by students, inflating false positives (e.g., BERT: 345 FP) and penalizing correct-but-divergent answers — evidence that rigid reference alignment undercuts fairness in open-ended assessment.

NLI exposes reference-based grading limits. Natural Language Inference analysis found that a large share of student responses fell into the contradiction category relative to reference answers, despite being semantically correct — confirming that students' short, diverse, own-word responses deviate from model answers and arguing for context-sensitive evaluation in Educational NLP.

Response features shape reliability. Student answers averaged only ~12 relevant words (23 total, ~10 stop words) with college-level readability (Flesch-Kincaid 14.79) and high lexical diversity, making short, sparse responses particularly challenging for reference-aligned models while GenAI models like GPTo1 handled them robustly.

Cost and scalability trade-offs. While GPTo1 was the only model considered deployable without human oversight, its proprietary API costs raise scalability concerns, leading the authors to recommend hybrid strategies combining advanced models with affordable options or human oversight in resource-constrained settings.

What this means for practice

• Assessment professionals. Prefer context-sensitive GenAI grading over reference-answer similarity for open-ended questions: GPTo1 recorded 11 false positives and 39 false negatives, while BERT recorded 345 false positives by penalizing correct answers that were phrased differently.
• Assessment professionals. Track false positives and false negatives by grade category instead of relying on accuracy, because even the strongest models showed slightly reduced recall on failing ('Fx') grades.
• Instructors. Where a proprietary high-performing model is unaffordable, keep a human in the loop rather than dropping to a similarity-based model — the authors recommend hybrid strategies combining advanced models with affordable options or human oversight in resource-constrained settings.
• Instructors. Require students to explain answers in their own words rather than list terms: bare term lists were consistently graded 'Fx' by the human evaluators, and set the same expectation for any automated model you adopt.
• Researchers. Benchmark against more than one human grader and report their disagreement: the two graders reached QWK = 0.95 overall but only QWK = 0.66 on question 13, showing the reference standard is itself graded with error.

Limitations

• The dataset comes from one blended Software Engineering course at a single institution (Constantine the Philosopher University in Nitra) — 1,885 responses to 24 questions from 110 students — and the authors name this single-domain focus as the principal obstacle to generalizing the results.
• Participation decayed across the course: about 110 students answered the early questions but only around 60 answered later ones, so question-level estimates rest on different samples.
• The human benchmark consists of two graders whose agreement was almost perfect overall (Cohen's Kappa = 0.84; QWK = 0.95), but it remains a two-rater standard in a single discipline.
• The study measures agreement with human grades, not learning: it offers no evidence that automated grading or feedback changes student outcomes, and the authors flag the proprietary API cost of GPTo1 as a scalability barrier.

Citation

Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models. International Journal of Artificial Intelligence in Education, 35, 3813–3846.