AIEd Knowledge BaseAIEd KB
On this page

Concept

Automated Assessment

created 2026-08-09 · updated 2026-11-11

Automated assessment — the use of AI to evaluate student work, from formative quizzes to high-stakes exams. Automated assessment spans multiple modalities — multiple-choice, short answer, essay, code, and performance-based evaluation — and ranges from direct automated grading to confidence-aware systems that report calibrated uncertainty alongside their scores.

Questions to Consider

  • Automated assessment ranges from multiple-choice scoring to essays, code, and performance-based evaluation. What's the biggest difference you'd expect between how reliably AI can grade a multiple-choice quiz versus a free-form essay — and why?
  • A core design idea here is 'confidence awareness': AI graders that report how certain they are, flagging low-confidence cases for human review rather than issuing a single unqualified score. How would a grade that said 'I'm 70% sure about this' change how you'd use or trust an automated score?
  • Research shows automated scoring can systematically disadvantage non-native speakers — the AI scores the language rather than the understanding. Why might an automated grader be especially prone to this kind of unfairness, even when it achieves high agreement with human raters overall?
  • One study found that how validation is done can inflate reported performance: a naive cross-validation method reported near-perfect results that dropped dramatically under more rigorous trial-independent validation. What does this cautionary lesson suggest about how you should read any claim that an AI assessment system 'works'?
  • The page argues calibrated confidence enables human-in-the-loop workflows, supports trust calibration, and strengthens measurement validity. If an automated system routed its most uncertain cases to a human reviewer, what would you want to know about how those cases are chosen before you trusted the split?
  • Automated grading is described as one of the most mature AIED applications, yet grading without useful feedback has limited educational value. How might a focus purely on producing scores — rather than usable feedback — change what students actually get out of an AI-graded assessment?

Introduction

Assessment modalities

Automated grading

Automated grading is one of the most mature and widely-deployed AI in education applications — AI systems that evaluate student work, from multiple-choice scoring to essay assessment and code review. Its grading modalities include:

  • Students separate feedback utility from evaluative authority: AlGhamdi (2026) documents a case where an AI score counted toward grades and students knew it — 13 computing students whose scanned handwritten work was evaluated by ChatGPT against a rubric covering clarity, organization, sentence correctness and conciseness. Every participant found the feedback useful, yet all separated that utility from authority, and several rejected the one-way, non-dialogical character of automated grading — "ChatGPT is not like people. You talk to them and explain excuses... I like normal teacher [to] grade my work." The study positions automated evaluation as acceptable for surface revision but not as a final assessor, with conditional trust mediated by instructor oversight.
  • Short answer grading: confidence-aware ASAG evaluate free-text responses, with confidence calibration critical — systems must know when grading is reliable. A scoping review of short-answer auto-marking in science (2017–early 2024) maps the field's course: Morley et al. found BERT-family models (base, RoBERTa, DistilBERT, SciBERT) dominated — used in 20 of 21 studies, peaking in 2021 — before GPT-based approaches were adopted via Prompt Engineering rather than fine-tuning from roughly 2022. Models augmented with domain data (textbooks, rubrics, further pre-training) consistently outperformed those without, yet few could justify marks in human-comprehensible terms and bias across demographic and linguistic groups was rarely examined — leading the authors to argue auto-markers should support rather than replace human examiners. A PRISMA-guided systematic review of 42 empirical grading and feedback studies (2023–2025) generalizes this caution across the field. LLMs match human raters on closed-ended tasks and short-answer questions but cannot fully replace human judgment on complex, open-ended, or subjective work requiring in-depth analysis or Creativity, and the highest grading effectiveness is achieved in hybrid systems that pair AI-driven grading with teacher oversight and verification (Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing Feedback).
  • Essay scoring: Automated Essay Scoring systems like anchor-based AES use prompting strategies to approach human-level reliability. RACES, a reward-aligned essay scorer, reported higher Quadratic Weighted Kappa and SimCSE similarity than evaluated pretrained and fine-tuned baselines on ASAP 2.0 without significance testing, and its reward-modeling feedback was LLM-generated proxy, leaving only the reference scores human-assigned (Li et al. (2026)). AIAWE extends automated evaluation to broader writing assessment.
  • Code review: Linux Bash grading and CS1 code review demonstrate automated assessment in computing education.
  • A prompt persona can collapse a grader, and fine-tuning can repair it: Habibullah et al. (2026) graded a practical computer vision exam (570 dual-graded students) under 171 configurations and a second machine learning exam (1,038 students) under 162 more. A short "strict grader" preamble drove 14 of 17 open-weight models out of the graded band (MAE ≥ 8), three stopping grading altogether; the damage traced to two credit-withholding sentences rather than tone or model scale, and its direction was exam-specific, improving seven models on the second exam. Light LoRA fine-tuning on roughly 3,900 pooled graded examples brought five small open models to parity or better with a human grader, so the vulnerability is a prompting artifact rather than a capability ceiling.
  • Formative integration: A-level science automation and CoTAL show how automated grading feeds into Formative Assessment cycles.
  • Coverage of feedback types is not alignment with teachers. Almousa et al. (2026) annotated paragraph-level feedback from three university writing courses into seven focus types; most models emit nearly every type, yet their distributions diverge from the teacher reference, with the educationally tuned LearnLM closest (Jensen–Shannon divergence 0.1344) and Mistral-7B furthest (0.2695). Prompting with category definitions or worked examples increased divergence for most models.
  • Bias and fairness: Language bias in physics scoring documents how automated grading can disadvantage non-native speakers — connecting to Bias Mitigation and Equity.
  • Benchmarking GenAI models for open-ended grading: Pecuchova, Benko & Drlik (2025) benchmarked eleven GenAI and sentence-embedding models against two expert human graders on 1,885 open-ended responses to 24 software-engineering questions. Only GPTo1 reached almost-perfect agreement (Fleiss' Kappa 0.82, low false positives and false negatives across grade categories), with Claude3 and PaLM2 close behind; context-sensitive GenAI models robustly handled short, diversely-phrased student answers, while reference-based embedding models (e.g., BERT's 345 false positives) systematically penalized correct-but-divergently-worded responses — evidence that model choice, not just grading format, shapes reliability for free-text open-ended assessment.
  • Prompt exemplars decide scoring agreement: Takahashi et al. (2026) automated hierarchical diagnostic reasoning (HDR), an applied-competence task in which learners find and explain deliberately embedded errors in a short case. Using GPT-4o with no fine-tuning, the team generated new problems and scored student descriptions: generated and human-authored problems carried the same internal consistency (Cronbach's α = 0.78) and showed no significant score-distribution difference across 100 participants, and the format's correlation with a reading-comprehension control stayed low (α = 0.36 and 0.41). Scoring accuracy depended on the prompt rather than the model — without worked examples Q2 precision fell to 0.42, while adding human-scored correct and incorrect exemplars lifted agreement to 98% or higher on every question (Q1 and Q2 at 100%) — against a measured human cost of 138 minutes of expert scoring for 117 responses plus 30 minutes of consensus.

A model's own reasoning mode is a second configuration lever: on 32 anonymized public-health master's exam submissions the same ChatGPT moved from weighted kappa 0.269 in fast mode to 0.718 in thinking mode, while Kimi reached 0.571 (Brevik et al. (2026)).

  • Multi-agent coding of teachers' open-ended responses, where the framework rather than the model carries reliability: Copur-Gencturk et al. (2026) extend automated assessment beyond student work to the coding of teachers' own content knowledge (CK) and pedagogical content knowledge (PCK), asking whether AI can reproduce the trained human coding of 268 U.S. middle school mathematics teachers' written responses about ratios and proportional relationships. A purpose-built multi-agent LLM, GradeOpt, assigns three GPT-4o-backed roles — a Grader that applies the human coding manual, a Reflector that diagnoses each disagreement with the human code, and a Refiner that folds clarification points back into the manual over up to six rounds — and matched trained human coders closely on CK (κ = .88 and .85) while reaching substantial agreement on the far more interpretive PCK items overall (κ = .68; weighted κ = .79). Two NLP baselines and a single-prompt GPT-4o model never exceeded κ = .39 or weighted κ = .55 on PCK, so the authors attribute reliability to refining instructions against the coding manual rather than to model scale, and recommend expert review wherever adjacent coding levels blur, since GradeOpt's residual errors were typically adjacent codes rather than wild misses.

  • Mixed-format exam grading (2026): Falahat, Das, Bhaumik & Thambi (2026) evaluated ChatGPT-5 against human faculty grading of a 21-item pharmacy exam (16 students) spanning multiple-choice, select-all-that-apply, fill-in-the-blank, listing, short-answer, and essay items. Concordance was substantial-to-near-perfect for objective item types (CCC 0.935–1.000 — aided by correct answers supplied during grading) but collapsed for listing (0.621–0.708), short-answer (≈0 to negative), and essay (0.341–0.854) items; providing a structured rubric did not consistently improve full-exam agreement (71.1% without vs 68.2% with). The study sharpens a methodological point that recurs across automated assessment — moderate percent accuracy frequently coexists with low concordance — so accuracy alone overstates reliability, and it recommends hybrid grading for complex, subjective, or high-stakes items.

  • Cross-family grade-band calibration on authentic marked essays (2026): Kerwat, Donaldson & Mahomed (2026) ran eight pre-specified configurations from four model families (GPT-OSS 20B, GPT-OSS 120B, Qwen 32B and Llama 3.1 8B, each at two temperatures) over the same locked cohort of 114 BAWE/Coventry essays carrying institutional Merit and Distinction labels, with one shared prompt that permitted four bands. Exact band agreement ranged from 18.4% to 54.4%: Llama 3.1 8B at temperature 0.5 led (54.4% exact, 98.2% within one band, MAE 0.474, near-zero bias) while GPT-OSS 120B was weakest (18.4%) and systematically undergraded, by an average of 1.316 bands at temperature 0.5, with Distinction accuracy trailing Merit accuracy throughout. The locked cohort contained no Pass or Fail references, so this is Merit–Distinction discrimination rather than reliability across the full grade range, and temperature was no cure — it helped one family descriptively, left another unchanged, and significantly worsened GPT-OSS 20B (39.5% to 29.8% exact). Two lessons transfer beyond the study: grade distance and directional bias reveal what agreement statistics hide, and model size is no proxy for assessment validity.

  • Rubric engineering for open-ended scoring in medical education: Olvet et al. (2026) asked whether GPT-4 could reliably score open-ended questions on pre-clerkship medical exams. After three iterations of human-driven rubric refinement at two US schools, inter-rater reliability with faculty reached substantial-to-almost-perfect agreement for three of four questions using analytic and holistic rubrics (weighted kappa up to 0.94), while the holistic-rubric item stalled at moderate (κw = 0.54). Error-pattern analysis showed discrepancies were traceable to both raters — GPT-4 over-scored when students offered multiple answers or used rubric-absent vocabulary, whereas faculty were often "overly generous" graders — leading the authors to recommend keeping humans in the loop (e.g., faculty scoring a subset to confirm accuracy). Because roughly 82% of US medical schools grade pre-clerkship work pass/fail, exact AI score agreement is often unnecessary for operational use.

  • HITL scoring for a large-scale national writing assessment (2026): Curi et al. (2026) scale prompt-based Large Language Models (LLMs) scoring to a real high-stakes Spanish writing exam (~5,000–6,000 responses/year), achieving 60–80% item accuracy and 90%+ output consistency across a 15-item analytic rubric, with a deterministic NLP checker replacing the LLM on the spelling item (100% consistency). Its distinctive contribution is decision-oriented validation over model-centric metrics: the AI's systematic under-grading bias, converted via IRT-and-Bookmark into proficiency levels, produces pass/fail discrepancies in 15.3–16.5% of cases — all of which the human-in-the-loop workflow routes to expert review, cutting responses needing full human scoring by at least 50% while preserving decision quality. This grounds hybrid grading in operational assessment rather than proof-of-concept datasets.

  • Per-dimension agreement shows where an assessor cannot detect change: SOPHIE 2.0 (Hasan et al., 2026) validated an LLM judge against the consensus of all human rating sources — standardized patients and third-party raters — on a three-dimension clinical communication rubric, reaching Pearson 0.759 and ICC(A,1) 0.746 from transcripts alone, inside the spread of the individual human raters rather than outside it. Its weakest dimension was Be Explicit (correlations of 0.414–0.598 across candidate judges), and that was also the only dimension whose scores did not move measurably across the two encounters (Δ = 0.014, p = 0.1204) — so per-dimension agreement can indicate in advance which dimensions an automated assessor is unable to show improvement on.

Confidence-aware assessment

A central design goal within automated assessment is confidence awareness: AI assessment systems that report calibrated uncertainty alongside their scores, rather than issuing a single unqualified prediction. A confidence-aware grader not only produces a grade or classification but also signals how certain it is, so that low-confidence cases can be flagged for human review and users can calibrate their Trust in the system. This is central to responsible automated assessment and connects closely to Psychometrically Aware AI and Trust Calibration.

How confidence is modeled in the knowledge base's research:

  • Fused confidence signals for short-answer grading: Confidence-Aware ASAG fuses model-based confidence signals (verbalized, latent, and consistency-based) with dataset-derived aleatoric uncertainty via Random Forest regression.
  • Confidence in multimodal student work: Confidence-aware assessment of student-drawn scientific figures extends confidence modeling to multimodal student responses.
  • Psychometric calibration of LLMs: Psychometrically aware AI advances the standard of aligning Large Language Models (LLMs) scoring with measurement theory, with calibration as a core requirement alongside Item Response Theory alignment.
  • Difficulty and response-time calibration: Programming-exam difficulty calibration repositions LLMs as auxiliary evidence sources whose difficulty estimates correlate with student pass rates.
  • Trait-adaptive essay scoring: PsyScore shows a psychometrically-aware framework can adapt essay feedback to learner traits.
  • Evaluating visual student work: DiagramIR back-translates LLM-generated math diagrams (TikZ) into an intermediate representation with deterministic checks, beating LLM-as-a-Judge on agreement with human raters and letting small models match large ones at ~10× lower cost — a scalable route to assessing non-text, diagrammatic student output.
  • Trust-gated inference in automated teacher assessment: Li, Yang & Fang (2025) place Monte Carlo dropout calibration directly in the scoring path, so that dropout variance above a learned threshold triggers a reject-and-refer output rather than a score, alongside adversarial debiasing that holds the fairness gap to 1.8% where baselines sit in the 6.4–8.2% range and an expected calibration error of 0.032 on TeacherEval-2023. Their ablation is the calibration argument in miniature: removing the trust-gated module drops inter-rater consistency to 78.6%, and the authors attribute a 41% reduction in human review workload to the gating — uncertainty handling as architecture rather than post-hoc reporting.
  • Explainability of rubric-based scoring: Bueno et al. show that model-agnostic SHAP attributions are more faithful and transferable than LLM-generated rationales for explaining rubric-based scores (e.g., classroom feedback quality), and propose deletion-based + cross-model tests as a principled way to evaluate any scoring model's explanations.

Why calibrated confidence matters:

  • Enables human-in-the-loop delegation: low-confidence cases route to a human reviewer — supporting human-in-the-loop workflows rather than blind automation.

  • Supports trust calibration: calibrated confidence lets users match their trust to the system's actual reliability, avoiding both over-trust and under-trust.

  • Improves measurement validity: confidence-aware scoring strengthens Educational Measurement and Assessment Validity by making uncertainty explicit.

  • Fairness and defensibility: flagging low-confidence cases for review reduces the risk of confidently wrong scores, especially for atypical or underrepresented responses.

  • Rubric generation and instructor-supervised grading pipelines: Mendonça et al. (2026) show that LLM-generated assessment rubrics (HARMOGEN-R) can match human-created rubrics for technical content within a ±5-point equivalence margin, with structured generation giving greater cross-model consistency. Cruz et al. (2026) evaluate an end-to-end GPT-4o grading pipeline where AI grades fell within 0.5 points of the instructor in 83% of 362 submissions (MAE 0.31) — best framed as a scalable supplement to, not a replacement for, instructor judgment.

Assessing teaching rather than work

Automated assessment usually scores what students produce, leaving the teacher's own performance to human observation. A 2026 integrative review turns the tools on teaching itself, mapping 21 studies drawn from 367 records in Scopus, Web of Science and ERIC (Dino et al., 2026). It followed Whittemore and Knafl's five-stage design with PRISMA reporting, appraised quality with MMAT 2018 — 89 of 105 criteria rated Yes — and built its keyword map in VOSviewer.

Four themes organize the corpus: Evaluation, models scoring teachers from student feedback and survey data; Engagement, tools sorting students by engagement level in online, on-site and hybrid settings; Expression, sentiment and topic coding of written student feedback; and Experience, sensor and video capture of observable classroom behavior. Teacher performance was the most common target, at 11 of 21 studies. The authors ran no meta-analysis, because methodological and outcome heterogeneity ruled out pooling, and 7 of the 21 studies had 100 participants or fewer — so the review maps a fragmented field rather than estimating how well AI judges teaching.

Quality and fairness

Automated assessment quality depends on Assessment Validity and Bias Mitigation. Language bias research shows that automated scoring can systematically disadvantage certain student populations. The criterion is itself fallible. Bozdag and Qiu (2026) mark GCSE physics scripts with an AI system and compare them against both official examination-board marks and experienced teachers' judgment, treating agreement with the board as a proxy for accuracy rather than accuracy itself.

  • AI disagreement as rubric diagnostics. Zhao et al. (2026) integrated an LLM grader into PrairieLearn so instructors see item-level disagreement with their own marking, then revise the rubric before scaling. AI–human agreement reached 99.4% on a single-item bug question and fell to 65% on a Markov-matrix problem with six valid solutions.

Deployment scenarios and what each one requires

How much alignment an institution should demand depends on what the system is allowed to decide. In a mixed-methods study of 761 authentic undergraduate essays at three UK universities, three frontier models with 27 prompt configurations each reached only 35–65 percent agreement with human markers on the degree band, and accuracy did not transfer between institutions — so the report treats candidate uses as distinct scenarios rather than one decision. Quality assurance of human marking, where AI marks in parallel and significant differences trigger human review, is the most conservative. A marking assistant, ranking or categorizing submissions by predicted quality or uncertainty, triaging complex cases, or expanding brief human comments into fuller feedback, is a genuine middle ground. AI as primary marker, with human review of a sample or of mark distributions, was judged acceptable only if system performance improves, and no stakeholder group endorsed AI as a sole marker.

Three requirements follow for any of them. Because alignment is model- and institution-specific, validation must be local and continuous — headline accuracy from elsewhere is not evidence of readiness. Because marks were compressed toward the middle, AI was least accurate at the grade boundaries and for the strongest and weakest submissions, which is where assessment decisions carry the most consequence. And because disagreement between an AI and a human mark reflects different judgment rather than mere error, discrepancies should trigger human interpretation rather than algorithmic override, with final authority over the mark retained by people. Adoption also carries AI Governance obligations the technical metrics do not capture: a right to explanation under GDPR Article 22 where automated decisions affect students, Equality Act duties once attainment level and language use are shown to influence marks, revised appeal processes for errors that are not easily caught, and an irreversibility risk — once staffing and investment shift, an institution may struggle to keep collecting the human marking it would need to return to.

Combining the three models did not remove that error: because their mistakes were correlated — the same direction on 63.4% of assignments — the Corsi et al. (2026) ensemble was statistically indistinguishable from the best single model, leaving the shared central-tendency compression in place.

Venetsanos (2026) supplies the criteria those requirements presuppose, arguing that what makes automation defensible is the epistemic status of the task rather than what the technology can technically perform. His framework limits AI to bounded verification of factual and procedural claims — all four of unambiguous documented criteria, direct comparison against established knowledge, no assessment of alternative valid approaches, and a single correct answer or pre-specified acceptable set must hold simultaneously, with ambiguity escalating to a human by default — and makes AI involvement at that level conditional on five non-negotiable principles that must all hold at once: the knowledge base must be assessor-curated and retrieval-grounded in module materials rather than the model's parametric knowledge; human assessors review every AI output before it reaches students and hold absolute override with unshared accountability; feedback must carry clear provenance and attribution; and security must be designed against adversarial use from the start, with input sanitisation for instructions hidden in white or small text, encodings, images or document metadata, since successful circumvention of an automated system spreads through student cohorts. The same paper is explicit that its principles are untested for simultaneous feasibility and that assessor time for curation, security infrastructure and high-frequency oversight may shift staff effort rather than reduce it — a caution that sits alongside the local-validation requirement above.

Security of AI-mediated grading

A red-team evaluation of an everyday grading workflow shows the attack surface that planning for adversarial use has to cover. Humble (2026) embedded five indirect prompt injections inside a synthetic essay that Microsoft Copilot (GPT-5.2) had graded fail in six of six baseline runs, iterating each across docx, pdf and htm files. Two strategies changed the grade with no visible warning — an instruction-manipulation and role-playing paragraph at 9 of 9 iterations (100%) and the same paragraph hidden behind an image layer at 17 of 18 (94%) — while a paragraph in small white text at the end of the document failed in all nine iterations, and file-metadata injections never worked, which the author attributes to metadata access being disabled in the version tested. The trust problem outlasted the exploits: after one pdf run detected an embedded instruction and stated it would grade only against the official assignment, re-running the same file raised the grade six more times with no warning, and a chat blocked by a detected attack was silently disabled rather than reported to the user. The author argues the resulting grades carry no validity claim in either direction, that an adversary needs only one working combination against layered Guardrails, and that the teacher remains the only real check on the output while the manipulation is designed not to be visible — connecting directly to the input-sanitisation and adversarial-security obligation Venetsanos sets out above.

  • The rubric itself is attack surface. Roy et al. (2026) show that a response written to satisfy a rubric's wording while violating its intent still earned a high score in 46.4% of verified attempts, even as inter-judge agreement stayed high. Their refinement operators — applicability gates, do-not-reward clauses, score caps — cut the rate to 36.0%.

Connections

Automated assessment connects to Assessment Validity (quality assurance), Formative Assessment (use context), Bias Mitigation and Equity (fairness), Teaching (how automation changes instructor work), and AI Feedback Quality (grading without useful feedback has limited educational value). Confidence-aware assessment is a specific mechanism within the broader agenda of Psychometrically Aware AI and a contributor to calibrated trust.

  • Large-scale AI grading of handwritten physics (2026): A multimodal model (GPT-5.5) graded 10,364 scanned pages across a national Physics Olympiad theory exam, selection camp, and university quantum-mechanics exam, achieving total-score correlations of 0.91–0.97 with official marks and recovering the same top-five Olympiad team. Revised page-by-page, evidence-location instructions improved agreement — evidence that multimodal AI can support high-stakes summative grading with careful rubric and prompt engineering (Large-scale AI grading of handwritten physics assessments: Score agreement and Olympiad team selection outcomes).

  • Agentic grading was reliable on discrete items, not on drawings. AlGhamdi (2026) had Claude Code grade 60 scanned handwritten exams (r = 0.97, ICC = 0.96), but weighted kappa fell from 0.98 on multiple choice to 0.65 on evaluation items, with 55% exact agreement on diagrams.

  • Selective automation for handwritten chemistry (2026): Cvengros & Kortemeyer graded a 296-student handwritten general-chemistry final page-by-page against rubric images with a multimodal, reasoning LLM, achieving high run-to-run reliability on total scores (ICC(A,1) = 0.967) and strong total-score agreement with TA grading (R² = 0.91). Because item-level reliability varied sharply by format — textual and chemical-reaction answers graded reliably while drawing and graphing scored worse than random — raw agreement was judged inadequate for high-stakes use. They operationalize human deferral through confidence filters (partial-credit thresholds, an IRT-based risk threshold, and problem-type exclusion) that convert raw AI scores into an accept/deferral policy, and show that false positives (AI crediting genuinely wrong answers) tend to go undetected since students rarely contest them — a concrete psychometrically grounded route to selective automation (Assisting the grading of a handwritten general chemistry exam with artificial intelligence).

  • Semi-open handwritten mathematics in a university exam (2026): Liu et al. graded a German undergraduate mathematics exam with GPT-4 across two extraction routes (pre-cut answer boxes versus whole pages), two optical-character-recognition routes and two rubric formats, reporting Krippendorff's alpha against human grades of 0.22-0.44 with accuracy 0.59-0.62 on the best workflows and per-problem accuracy spanning 0.27-0.91 - not acceptable for high-stakes summative use. Itemising multi-criteria grading rules generally did not improve agreement (it helped the partial-credit problem), free grading without a rubric was far worse (accuracy 0.50-0.51), and a probabilistic confidence filter meant to flag grades for human re-checking did not significantly improve accuracy while wrongly flagging 7-41 percent of responses as reliable. The study's transferable lesson is procedural: scan and transcribe the whole exam sheet with markers rather than cropping answer boxes, because students write across box borders; rewrite grading rules written for teaching assistants, since the model attempted unrequested derivations from a rule meant for a human; and note that the human-grading rules themselves paraphrased at an alpha of 0.88 among five variants, so prompt robustness is not the bottleneck.

  • LLM comparative judgment for writing screening (2026): Mercer & Reed used seven LLMs as pairwise comparative judges to score informational writing from 1,208 students in Grades 3–6 across three screening occasions. LLM-CJ scores correlated strongly with analytic rubric scores (r = .67–.73, strongest for Gemini-3.1 Pro) with AUC .82–.86 for proficiency; findings were stable across models, with little validity gain from costlier, more capable ones, while averaging three writing samples improved accuracy — sampling breadth mattered more than model choice. Predictive-bias patterns for multilingual learners matched human rubric scoring, supporting LLM-CJ as an efficient, low-cost screening approach (Validity of Large Language Model Comparative Judgment for Universal Writing Screening).

  • Quality-dependent alignment with instructor grading (2026): Comparing ChatGPT, peers, and an instructor grading the same undergraduate group projects, Usher & Faraon found ChatGPT's alignment with the instructor improved as project quality increased — its largest overestimation was for low-quality work (≈ +14 points), shrinking to ≈ +2.5 points for high-quality projects. ChatGPT also graded higher on average than both peers and the instructor, a grade-inflation tendency that undermines its reliability as a standalone summative grader, especially for weaker submissions.

  • Small-corpus agreement statistics can mislead deployment decisions. Scoring 60 marketing posts (15 students plus 15 researcher-authored low-quality anchors) against two independent human raters gave absolute agreement ICC(2,1) of .435 for the LLM, .266 for an equal-weight rule-plus-LLM hybrid and .091 for deterministic rules, with MAE of 6.28, 10.53 and 17.22 points respectively on a 0-100 scale; the hybrid was significantly worse than the LLM alone (paired ICC difference -.169, 95% CI [-.260, -.106]). Adding anchors raised inter-rater ICC from .338 to .902 and LLM agreement from .435 to .846, and a near-empty post was awarded 75 against a human mean of 30.5, showing that a single degenerate response can dominate a small evaluation. (Agreement and error in automated scoring of student marketing posts)

  • Structured LLM grading can look analytic without being diagnostic (2026): Scoring 3,041 responses to 50 computer-science questions, three commercial LLMs produced rubric sub-dimensions that were near-redundant (r = 0.82–0.99, VIF up to 45) and named Misconceptions about AI in only 4.8–7.4% of comments, against 15.0% for instructors (Zhao et al., 2026).

  • "A single kappa is the case for a grader." It is one check, and a benchmark of 22 prompted models on three public corpora shows why. Every model met probes whose right answer is known: a polite request for full marks, a forged grading instruction, a blank answer, a nonsense word, a bare objection. Fourteen were flagged for gaining more than 5% of the scale from a note, up to 83.2% on code, and seven credited blank or meaningless answers, so the agreement figure and the failure tests have to be read together. A deployed grading system has one more input to test: sharing a learner model's estimate with the grader shifted marks by 0.28 to 0.75 points on identical work, and the ablation that found it left rater agreement almost unchanged.

  • Tagging agreement depends on the framework. Against faculty tags on 62 pharmacy questions, an AI curricular-mapping tool reached kappa 0.69 for disease states and 0.57 for Bloom's, but only 0.24 for COEPA subdomains, and the comparison allowed faculty multiple tags against the tool's single choice (Tag, You Are It: Evaluating AI and Faculty Alignment in Question Categorization Across Four Competency Frameworks).

Common Misconceptions

  • "An automated score is a single reliable number." A practical computer vision exam (570 dual-graded students) was graded under 171 configurations, and a second machine-learning exam under further ones — the configuration drove the score. A grade presented as a number conceals the configuration that produced it.
  • "The judge is the source of unreliability." On 378 fixed-dice rulings, a re-run changed 2.9 to 6.1 per cent of outcomes under a semi-rigid method and 13.5 to 15.9 per cent under free adjudication, while swapping the judge AI added only about 3 points (Epstein, 2026) — the procedure, not the judge, sets reproducibility.
  • "Confidence reporting is a nicety." A score that said it was 70% sure would change how it is used, and the absence of that information forces the reader to treat all scores as equally certain.
  • "Short-answer auto-marking is a settled area." A scoping review of short-answer auto-marking in science (2017 to early 2024) documents the field's history, which is a record of what has been attempted rather than of what has been validated.

Connected Concepts

Connected Articles

Connected FAQs

Metadata

Intended audienceInstructors
Technologies and techniquesLarge Language Models (LLMs)
Assessment and measurementAssessment, Assessment Validity, Automated Assessment, Automated Essay Scoring, Formative Assessment
Equity, ethics, and responsible useBias Mitigation
Foundations of AI in educationTeaching
SourceHistoryEditChat

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.