Automated Essay Scoring

Automated Essay Scoring (AES) — the use of AI to evaluate and score written essays, spanning traditional statistical approaches, fine-tuned language models, and increasingly accessible LLM-based prompting strategies. AES research in this wiki covers scoring accuracy, fairness and bias, psychometric validity, and practical accessibility for educators.

Automated Essay Scoring has a long history in educational technology, from early statistical models to modern LLM-based approaches that can evaluate essays holistically without large pre-scored datasets. The key tension in AES research is between accuracy and accessibility — while fine-tuned models achieve strong results, they are resource-intensive and impractical for most educators.

Key research themes

Prompting-based AES has emerged as the most accessible approach. The Choi et al. anchor paper study shows that including exemplar essays in prompts brings LLM-human agreement close to human-human reliability, with GPT-4o mini achieving comparable results to GPT-4o at lower cost. This connects to broader Prompt Engineering research and makes AES feasible for teacher use.

Psychometric and trait-level scoring moves beyond holistic scores. PsyScore provides a psychometrically-aware framework for trait-adaptive scoring with ZPD-grounded feedback. ICLE++ models fine-grained traits for holistic essay scoring, advancing the precision of automated evaluation.

Bias and fairness is a critical concern. Feser & Tschisgale found that AI scoring systematically underestimates students from linguistically diverse backgrounds, highlighting the need for Bias Mitigation and Equity considerations in AES deployment.

L2 and self-referential assessment explores non-native writing contexts. Profile-based L2 assessment uses a self-referential approach comparing student writing to their own prior work rather than native-speaker norms.

Connections to related concepts

AES sits at the intersection of Automated Assessment, Writing Education, and Generative AI. It connects to Formative Assessment when used for feedback rather than grading, to Feedback Loop when integrated into iterative writing processes, and to AI Literacy when educators understand and calibrate AES tools. The Assessment Validity and Educational Measurement concepts are essential for ensuring AES scores are meaningful and fair.