AIEd Knowledge BaseAIEd KB
On this page

Concept

Automated Essay Scoring

created 2026-08-09 · updated 2026-11-10

Automated Essay Scoring (AES) — the use of AI to evaluate and score written essays, spanning traditional statistical approaches, fine-tuned language models, and increasingly accessible Large Language Models (LLMs)-based prompting strategies. AES research in this knowledge base covers scoring accuracy, fairness and bias, psychometric validity, and practical Accessibility for educators.

Questions to Consider

  • Automated Essay Scoring has moved from traditional statistical models to LLM-based prompting. A key tension on this page is between accuracy and accessibility — fine-tuned models score well but are impractical for most educators. Which would you prioritize in your own context, and why?
  • A major finding is that simply including exemplar essays in the prompt brings LLM-human agreement close to human-human reliability — and a cheaper model can match a more expensive one. What does this suggest about how much of AES quality is the model versus how it's prompted?
  • The page warns that AI scoring can systematically underestimate students from linguistically diverse backgrounds. Before reading, if you saw an AI give a lower score to a non-native speaker's essay, would you have assumed it was a 'bias problem' or just 'the score'? What would change how you respond?
  • A self-referential approach assesses L2 writers by comparing their writing to their own prior work rather than to native-speaker norms. How does the choice of comparison baseline change what a score means — and which students might it treat more fairly?
  • AES intersects with formative assessment when used for feedback rather than grading. When would a machine's feedback on an essay be genuinely useful to a developing writer, and when might it flatten the kinds of qualitative feedback a human editor would give?

Introduction

Automated Essay Scoring has a long history in educational technology, from early statistical models to modern LLM-based approaches that can evaluate essays holistically without large pre-scored datasets. The key tension in AES research is between accuracy and accessibility — while fine-tuned models achieve strong results, they are resource-intensive and impractical for most educators.

  • Zhang et al. RACES uses reward alignment to make LLM essay scoring both accurate and consistent, addressing a core AES validity concern.

Xu, Lu and Abdi (2026) review twelve studies comparing AI and human scoring of EFL academic writing and report moderate alignment overall, strongest on surface linguistic features and weakest on organization, coherence and argument development, with fairness rarely examined.

In upper-elementary response-to-text writing, Correnti et al. (2026) find that automated feedback on students' use of evidence improved feature scores, and that revision quality depended on whether teachers took the feedback up in discussion, not on the system alone.

Key research themes

Prompting-based AES has emerged as the most accessible approach. The Choi et al. anchor paper study shows that including exemplar essays in prompts brings LLM-human agreement close to human-human reliability, with GPT-4o mini achieving comparable results to GPT-4o at lower cost. This connects to broader Prompt Engineering research and makes AES feasible for teacher use.

Psychometric and trait-level scoring moves beyond holistic scores. PsyScore provides a psychometrically-aware framework for trait-adaptive scoring with ZPD-grounded feedback. ICLE++ models fine-grained traits for holistic essay scoring, advancing the precision of automated evaluation.

  • Reasoning traces as the explanation. Xu et al. (2026) fine-tune Qwen models to reason through the four IELTS traits before scoring, then apply PPO on examiner preferences. Average QWK rose from 0.5253 (7B, CoT only) to 0.5410 (72B), while feedback BLEU rose from 0.2371 to 0.2873 with PPO and CoT; scale alone added little, at 2.5% BLEU from 7B to 72B.

  • Auditability as a scoring target. Xin (2026) builds an ASAP pipeline whose feedback layer attaches essay evidence to each weak trait and exposes a teacher review flag. Macro QWK was 0.738 against 0.739 for its best pure scorer, while rubric-grounded feedback reached a Trait Recall@2 of 0.829 and a valid evidence rate of 0.912 where rubric-only feedback cited nothing.

  • Dimension-level trait scoring is bounded by the data behind each dimension. Firdausi et al. (2026) trained one classifier per dimension of Ennis's FRISCO Critical Thinking framework on 106 Indonesian eleventh-grade physics essays (83 for training, 23 for testing, with three teacher raters agreeing at kappa 0.78–0.84 on every dimension). Agreement concentrated on expression rather than reasoning: quadratic weighted kappa reached 0.763 for Clarity and 0.728 for Situation but only 0.374 for Focus, 0.332 for Reason and 0.227 for Inference. The ordinal signal was manufactured rather than observed. Synthetic level-1 essays (built from templates, cut to 20–40% of their original length, with keywords removed) plus LLM paraphrase lifted Focus by +0.346 kappa off a 0.028 baseline — and one configuration reached 0.652 accuracy with near-zero or negative kappa, — so accuracy is not a usable selection criterion in ordinal AES.

Bias and fairness is a critical concern. Feser & Tschisgale found that AI scoring systematically underestimates students from linguistically diverse backgrounds, highlighting the need for Bias Mitigation and Equity considerations in AES deployment.

Teachers want the score presented as reviewable, not final. Nine experienced Brazilian high-school writing teachers evaluated an AIED-Unplugged essay-scoring app and rated it highly — median SUS 92.5, TAM medians of 7 for usefulness, ease of use, relevance and intent to use, 6 for result quality. They volunteered, unprompted, that the report should carry an on-screen warning that it is not fully reliable and must be reviewed before a student sees it; one teacher read that obligation as a positive feature (Ribeiro et al. (2026)). They asked for more detail, not less: per-competency diagnostics, a global summary, a field for their own comments, and concrete examples of how to improve. Where validity research argues about agreement rates, these users asked the interface to encode the division of judgment between system and teacher.

Disagreement with human graders is systematic, not random. Out-of-the-box GPT and Llama models agreed only weakly with human scores (QWK ≈ 0.17–0.28 against 0.72 between two human raters) and biased in a quality-dependent direction — higher for short, underdeveloped essays and lower for long, strong ones carrying minor surface errors (Mathew et al. (2026)).

L2 and self-referential assessment explores non-native writing contexts. Profile-based L2 assessment uses a self-referential approach comparing student writing to their own prior work rather than native-speaker norms.

Interpretability and feature weighting opens the blackbox of how LLMs actually score. Wang et al. (2026) compared three LLMs (Qwen, GPT, Gemini) with human raters on non-native English essays across sixteen textual features, finding strong overall alignment but distinct weighting: LLMs emphasized grammatical accuracy, lexical sophistication, and syntactic complexity, while human raters prioritized content completeness and visual presentation. Critically, LLMs shifted their weighting by proficiency level — placing more weight on language errors for low-proficiency students and increasingly rewarding linguistic sophistication for high-proficiency students — whereas human raters maintained a more stable framework. Liu, Ye, and Yan (2026) extend this with a five-model evaluation framework (GPT-4.1, Llama 4 Maverick, Gemini 2.5 Flash, Claude Sonnet 4, DeepSeek R1) on 60 long essays, using causal discovery to reveal distinct evaluative heuristics: most models prioritized lexical precision and fluency, while others emphasized syntactic complexity or cross-domain integration, and some showed inconsistency, score compression, or systematic underestimation. Together these studies establish that AES validity depends not only on overall agreement but on how models weight features and whether that weighting is stable across learner subgroups — directly informing Assessment Validity and Bias Mitigation auditing.

Item-type boundaries and the limits of essay grading. In a mixed-format university exam, Falahat, Das, Bhaumik & Thambi (2026) found ChatGPT-5's concordance with faculty was substantial-to-near-perfect on objective items (CCC 0.935–1.000) but dropped sharply on open-ended responses — near-zero to negative for short-answer and only 0.341–0.854 for essay questions — and a structured rubric did not consistently improve essay agreement. This bounds AES validity: model fluency helps on well-specified items but does not carry over to holistic essay scoring, where contextual interpretation of partial-credit responses still favors human judgment.

Issue-type boundaries in diagnostic agreement. Yao and Fan (2026) used DeepSeek-R1 to label specific writing issues before revision and compared its labels with teacher diagnosis across three cycles (mean F1 = 0.768, SD = 0.037). Agreement was highest on locally cued issues such as vocabulary word choice (F1 = 0.824) and lowest where the issue turns on a claim–evidence relation (source use and evidence, 0.579), so agreement in a diagnostic role depends on the issue type, not only on the model or the prompt. A skewed rubric distribution can make aggregate scoring metrics mislead. Automated evaluation of a rubric dimension may be a classification task with a lopsided label distribution rather than a continuous score: marking reflection level 0-3 in 1,954 Hungarian student essays, 68% of essays sat at level 3 and 1.8% at level 0, and the best shallow configuration (SMOTEBoost with Qwen3 embeddings) reached an overall score of 0.7176 averaged over accuracy, F1 and ROC-AUC against 0.6872 for the best fine-tuned transformer — yet one RidgeClassifier configuration scored 0.7131 ROC-AUC at only 0.5162 accuracy, and class balancing lowered every metric (accuracy falling from 0.6672 without it to 0.6408 with it), so the authors read model choice as a question of which metric matters, with transformers earning their place through minority-class sensitivity rather than aggregate agreement (Csibi et al., 2026).

High-stakes deployment evidence. Field evidence from a real high-stakes deployment (Uruguay's Acredita EB, 2024–2025; Curi et al.) shows prompt-engineered GPT-5 reaching 60–80% agreement with trained human raters across a 15-item analytic Spanish rubric, roughly five percentage points below human inter-rater agreement for most items and never more than 15 points below, with run-to-run consistency above 90% for almost all items. Vocabulary, syntax and spelling were the weakest dimensions, and the spelling item had to be handed to a deterministic grammar checker (LanguageTool, 68% accuracy, 100% consistency) because token-level orthography is where the model is least stable. Prompts built for one exam edition transferred to the next with only topic-specific edits, and the AI was systematically stricter than humans — under-grading rather than over-grading. For AES design, the lesson is that prompting-based scoring can approach human agreement even in a national exam, while its remaining weakness sits at the level of low-level language conventions rather than holistic writing quality.

Token-level confidence, probability-weighted scoring, and ensembling each improved agreement on divergent-thinking scoring — the stacked techniques reached r = 0.846 (RMSE = 0.4848), and routing only the least confident 20% of responses to human review cut manual effort by roughly 80% (Organisciak & Acar (2026)).

The human Benchmark, and what it can and cannot license. The OpRaise study is the strongest test in this knowledge base of whether LLM marking is ready for routine use, and its answer turns on the benchmark rather than the model. It compared three frontier systems (Claude Opus 4.6, GPT-5.4, Gemini 3 Flash), each under 27 prompt configurations crossing rubric specificity, calibration and scoring strategy, against the moderated marks of 761 authentic Psychology essays from 125 students at three UK universities. Human marks were adopted as ground truth because academic judgment is the socially accepted standard, and the authors note that human markers agree only moderately with each other — which bounds how strong AI–human agreement could reasonably be demanded to be. Against that benchmark, agreement on the UK degree band ranged from 35 to 65 percent by institution (63 percent at Cambridge, 53 percent at Nottingham, 35 percent at Manchester Metropolitan) and did not transfer between them, so the report's central recommendation is local validation on an institution's own Assessment materials. Two findings generalize beyond this corpus. First, reliability and agreement pull apart: every model re-marked nearly identically (ICC 1.00, 1.00, 0.97) and the models agreed with each other more closely than with humans (three-model ICC 0.91), yet all three agreed on the degree band for only 56 percent of submissions — self-consistency is not validity. Second, AI marks compressed toward the middle of the scale (a compression score of 0.47–0.82, with the crossover where AI and human agree on average sitting in the upper 50s to low 60s), making AI least accurate precisely at the grade boundaries that separate Firsts from Upper Seconds and passes from fails. Vocabulary range, connectives, sentence complexity and text length predicted AI marks with small but significant effects while their relationship with human marks was broadly negligible — a direct demonstration of the linguistic-bias concern, and one that no prompting strategy tested removed.

AES as a training signal rather than a judge. Scoring models are usually studied as evaluators, but SWIM uses one as a reward function. Do, Kontak and Sachan (2026) freeze a multi-trait AES verifier and score it against generated student essays, turning the predicted trait profile into a dense per-essay reward (the mean trait-normalized distance from the target profile) for GRPO, deliberately not the Quadratic Weighted Kappa metric itself. QWK is defined over a batch of target-prediction pairs and is not a per-sample signal, while exact-match rewards are too sparse in the multi-trait setting. The gains were re-checked against a DeBERTa scorer and an evaluation-only verifier the policy never trained against (0.598 versus 0.479 for SFT, and 0.647 versus 0.501), which is the check that separates genuine proficiency control from adaptation to the reward model. For AES research this is a second validity demand: a scorer used as a training target is being optimized against, so its own trait weighting and language biases propagate into everything the generator learns. The same feature-weighting asymmetries (grammatical accuracy, lexical sophistication and syntactic complexity weighted more heavily by models than by human raters) documented elsewhere on this page now shape training rather than only marks. Scoring and feedback want different methods, so a system should separate them. Labib et al. (2026) built one TOEFL argumentative-essay system as three modules — scoring, surface-level feedback on grammar and mechanics, and deep-level feedback on organization, coherence and argumentation — instead of a single model doing all three jobs. What wins on one module loses on another: supervised fine-tuning of GPT-4o on 120 of a proprietary set of 480 benchmark-scored essays, tested on the remaining 360, reached a quadratic weighted kappa of 0.84 and RMSE of 0.44 on the 0-5 scale, ahead of the Wang and Gayed (2024) baseline (QWK 0.78, RMSE 0.57). Yet the same fine-tuning produced unusable output for the feedback modules, where direct prompting of Claude 3.7 was what generated teacher-approved comments.

AES sits at the intersection of Automated Assessment, Writing, and Generative AI. It connects to Formative Assessment when used for feedback rather than grading, to Feedback Loop when integrated into iterative writing processes, and to AI Literacy when educators understand and calibrate AES tools. The Assessment Validity and Educational Measurement concepts are essential for ensuring AES scores are meaningful and fair.

  • Agreement, error and what a hybrid scorer adds. In a small open-ended marketing-writing corpus the LLM out-scored both deterministic rules and an equal-weight hybrid on absolute agreement with human raters (ICC(2,1) .435 versus .266 and .091), with the hybrid significantly worse than the LLM alone, while score dispersion and a single near-empty response showed how strongly such estimates depend on corpus composition (Agreement and error in automated scoring of student marketing posts).

Open systems are part of this picture too: AiAWE scores argumentative essays with a LoRA-adapted Gemma-3-27B-it, reaching QWK 0.828 and agreement within ±0.5 of the human score on 90.56% of 360 evaluation essays — and reports that model scale did not reliably predict downstream performance under LoRA adaptation (AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models).

  • "The prompted model that agrees best is the strongest essay scorer." On ASAP 2.0 the published agreement between the two expert raters is κ 0.745, and no prompted model's point estimate reached it: the 22 ran from 0.155 to 0.727, and for 18 the whole interval lay below. A fine-tuned DeBERTa scorer reached 0.809 and a rule that counts only essay length reached 0.636, above 16 of 22 models, though every model kept agreement beyond length. The baseline to beat is the trained scorer, not the model down the road.

Common Misconceptions

  • "Automated scoring results reflect student writing." The ordinal signal can be manufactured rather than observed — synthetic level-1 essays built by template generation, degradation to 20-40% of original length and keyword removal. A system validated on constructed extremes may never have been tested on the distribution it will meet.
  • "Aggregate accuracy is the metric to report." A skewed rubric distribution can make aggregate scoring metrics mislead: automated evaluation of a rubric dimension is often a classification task with a lopsided label distribution, where high accuracy and useful discrimination separate.
  • "Field deployments have not been evaluated." Field evidence exists from a real high-stakes deployment (Uruguay's Acredita EB, 2024), which is the strongest form of evidence in this literature and the one most studies of scoring accuracy never supply.

Connected Concepts

Connected Articles

Metadata

DisciplineWriting Education
LevelHigher Ed, K 12
Technologies and techniquesGenerative AI, Large Language Models (LLMs), Prompt Engineering
Assessment and measurementAssessment, Automated Assessment
Foundations of AI in educationAI Literacy
SourceHistoryEditChat

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.