Risks of Using Large Language Models in Grading: LLMs and Humans Prefer LLM-Generated Writing Over Human's but LLMs Show a Stronger Systematic Bias

Synthesis: Mak (University of Warwick) asks whether an LLM can grade student work fairly when the work it is judging may itself have been written by an LLM. Study 1 had four evaluators — GPT-4.1, GPT-5, Gemma3:1b and Qwen3:4b — score 1,426 authentic undergraduate psychology dissertations alongside GPT-4.1 and GPT-5 rewrites of each and fully GPT-generated versions prompted from the student abstracts. Every evaluator gave student-authored dissertations the lowest scores, rewrites intermediate scores and fully generated dissertations the highest, with effects as large as d = -1.85. Study 2 replicated the graded pattern on 50 abstracts for three of the four model evaluators and, in the paper's key contribution, ran a preregistered experiment with 300 human raters. Humans rated student-authored and GPT-rewritten abstracts as equivalent and preferred fully GPT-generated abstracts only moderately (d = 0.58), while the equivalent model contrasts exceeded d = 0.85, with z-tests confirming the preference was significantly stronger in LLMs. Because fully generated texts beat mere rewrites, the bias is not purely stylistic; it rewards characteristic LLM organisation and reasoning. The implication is that automated grading may advantage AI-assisted work and needs bias auditing before deployment.

Key Findings

1. Study 1 used a very large authentic corpus. The student corpus contained 1,426 final-year psychology dissertations submitted by 1,426 unique students across ten academic years (2015/6 to 2024/5), averaging 5,532 words and roughly 7.89 million words in total, each conforming to a standard six-section empirical structure.
2. Three text variants were compared. GPT-4.1 and GPT-5 each rewrote every student sentence (7.79 million and 7.92 million tokens), preserving content while changing style; separately, GPT-4.1 and GPT-5 generated entire dissertations from the student abstracts (7.45 million and 7.89 million tokens), differing in both style and substantive content.
3. All four LLM evaluators showed the graded preference. Mean scores for student-authored dissertations were lowest under every evaluator (GPT-4.1 71.3, GPT-5 67.7, Gemma3:1b 73.8, Qwen3:4b 75.7), with fully GPT-generated versions highest (up to 90.1 under Qwen for GPT-5-generated text); every GPT corpus outscored the student originals at p < .001.
4. The Study 1 effects were often large. Effect sizes across the 16 student-versus-GPT comparisons ranged from d = -0.12 to d = -1.85, and 10 of the 16 exceeded d = 0.8; the student-versus-fully-GPT-4-generated contrast was the largest for GPT-4.1 (d = -1.85) and Qwen (d = -1.31), and student-versus-fully-GPT-5-generated was largest for Gemma (d = -1.21) and Qwen (d = -1.83).
5. Fully generated texts beat rewrites, so the bias is more than stylistic. Rewrite-versus-full comparisons were significant at p < .001 for seven of eight contrasts with effects from d = -0.36 to d = -1.61, the only exception being GPT-5's non-significant d = -0.04 for its own rewrite against fully generated text.
6. Study 2 replicated the pattern on short abstracts. Fifty abstracts (five per academic year, drawn from the 804 whose length fell between 160 and 180 words) were each paired with a GPT-5 rewrite and a fully GPT-5-generated version; GPT-4.1, GPT-5 and Qwen again scored student texts lowest, by an average of 12.7 points versus rewrites (d = -0.64 to -1.35) and 16.9 points versus fully generated versions (d = -0.87 to -1.61).
7. Human raters behaved differently. In the preregistered experiment (300 Prolific adults, M-age = 37.1, 155 assigned to the Rewritten condition and 145 to the Fully GPT-Generated condition), humans rated student-authored and GPT-rewritten abstracts as equivalent (beta = 1.45, t = 1.68, p = .093, d = 0.10) — a direct contrast with the LLM evaluators, which all preferred the rewrites.
8. Humans also preferred fully generated abstracts, but far more weakly. In the Fully GPT-Generated condition student abstracts averaged 8.9 points lower than GPT-generated ones (beta = -8.85, t = -9.90, p < .001, d = -0.58), versus model effects consistently beyond d = 0.85 for the same contrast.
9. The gap was statistically tested, not eyeballed. A significant Condition by Authorship interaction (p < .001) was followed by z-tests on the effect sizes: the GPT preference was significantly stronger for GPT-4.1 (z = 5.08, p < .001), GPT-5 (z = 5.48, p < .001) and Qwen (z = 2.56, p = .010) than for humans.
10. Gemma was the dissenting evaluator in Study 2. Gemma3:1b, the smallest model, showed no significant student-versus-rewrite difference (p = .429, d = 0.14) and no significant student-versus-fully-generated difference (p = .070, d = -0.35), unlike its consistent behaviour in Study 1; the author speculates that abstract-length texts gave it fewer distinguishing cues.

The two studies and how they were designed

Study 1 asked whether LLMs give higher marks to GPT-generated work than to human writing, and whether any such preference is stylistic or extends to the content GPT produces. The design exploited a natural three-way contrast within the same pool of dissertations: the original student-authored texts, sentence-level GPT rewrites that preserved the ideas and arguments while changing the prose, and fully generated dissertations built only from each student's abstract, so that content as well as style was machine-produced. Four evaluators spanning proprietary frontier models (GPT-4.1, GPT-5) and smaller open-weight models (Gemma3:1b, Qwen3:4b) graded every text using the departmental score descriptors, two authentic exemplar dissertations at the top and bottom of the 2024 range, and instructions to use the full 0-100 range.

Study 2 moved to a smaller unit of analysis to make human comparison feasible: 50 student abstracts sampled within a narrow 160-180 word band, with GPT-5 rewrites and fully GPT-5-generated counterparts length-matched to their student originals within 5 words. The reason for the shift is stated plainly — full dissertations typically exceed 5,000 words, which would have made large-scale rating by non-experts impractical. The same four models graded the abstracts, and 300 preregistered human participants rated them in a 2 × 2 mixed design, with Authorship (Student versus GPT) varied within participants and Condition (Rewritten versus Fully GPT-Generated) between participants.

Isolating self-preference from length, quality and prompt

The paper stacks three controls against the most obvious alternative explanations. Length is addressed directly in Study 2, where abstracts were sampled within a fixed word range and rewritten or generated texts were adjusted to within five words of their student counterparts, so score differences cannot be attributed to verbosity or concision. Prompt variation is addressed by design: Study 1 instructed a psychology professor to apply marking descriptors, while Study 2 instructed a layman reviewer to judge general writing quality on the same 0-100 scale, using wording that could be given identically to models and people — and the graded preference survived the change.

The hardest rival explanation, that GPT texts are simply better written and deserve higher marks, is what the human comparison exists to test. If human raters had shown the same preference, the LLM grading pattern would be more parsimoniously read as a quality difference rather than an evaluator bias. Because humans rated student and rewritten abstracts as equivalent while GPT-4.1, GPT-5 and Qwen all clearly favoured the rewrites, the residual gap is evidence of a self-preferential bias in the models rather than an objective quality advantage.

Finally, the rewrite-versus-full contrast separates style from substance. A purely stylistic preference would have scored rewrites and fully generated texts alike, since both are GPT prose; instead the fully generated versions were almost always rated higher. The author reads this as models also rewarding the organisation, argumentative structure, coherence and reasoning patterns characteristic of their own generated content — a preference for deeper properties rather than surface word choice. Because the generative models used as evaluators (Gemma, Qwen) also marked text they had not produced, casual memorisation of one's own outputs is ruled out as the explanation.

Separating the human baseline from the model bias

Study 2's central comparison is a numerical separation of human judgement from model judgement. Human raters did find a real difference, just a modest one: 8.9 points on a 100-point scale, d = 0.58, which the author treats as consistent with a genuine perceived quality difference — fluent, well-organised machine abstracts may simply read better to a general reader. The model evaluators, by contrast, produced effects that were both larger and differently distributed: they penalised student work at essentially every level, including against rewrites whose ideas were identical to the student originals, and their preference for fully generated text reached d = 1.61.

The validity reading follows from that asymmetry. If both people and models reward machine writing, the human behaviour can be folded into ordinary quality judgement; the model behaviour cannot, because it appears even where content is held constant. That is what makes the bias systematic rather than substantive: models are not only tracking quality, they are tracking similarity to their own output distributions. The paper's proposed mechanism is familiarity — text produced from the same learned regularities that a model reproduces may appear more predictable and thus be judged better, so human writing that deviates from those regularities is penalised despite comparable quality.

What it means for automated grading and AI detection

For automated grading, the consequence the author draws is fairness and equity, not efficiency. Students who use generative AI heavily may receive higher grades than equally capable peers who work independently or lack tool access, which means grades could partly measure conformity to LLM-centred writing patterns rather than knowledge and skill, and could actively incentivise AI-assisted writing. The recommendation is not a technical fix but a gate: LLM grading systems should face rigorous bias auditing and validation before or instead of deployment, and institutions should exercise extreme caution until systems can be shown to grade human- and AI-authored work without systematic bias.

The findings also cut against a common detection-adjacent proposal — using one LLM to judge whether another LLM wrote a text. If an evaluator systematically rates machine prose higher even when content is held constant, then a model's confidence about authorship and its judgement of quality are entangled; a tool that prefers machine text is a poor instrument for remaining neutral about it. The same failure mode matters for the detection and integrity workflows that increasingly sit beside AI grading, and it compounds concerns raised elsewhere about whether detector verdicts are reliable enough to carry disciplinary weight.

Limits, and how far the result travels

The dataset, though large, is narrow: higher education psychology dissertations and abstracts from a single UK department, in English only, with writing conventions and Assessment criteria that need not hold in other disciplines or languages — the author expects the magnitude and maybe the nature of the bias to differ elsewhere. The human sample is a general-population Prolific pool with at least undergraduate education rather than subject experts, and its Study 2 task was deliberately restricted to lay judgements of writing quality, so the human baseline is a quality-perception baseline rather than an expert marking baseline.

Study 2 used GPT-5 exclusively for generation, and its abstract-level analyses rest on 150 texts, giving weaker power than Study 1; the rewrite-versus-full comparisons there would not survive correction for multiple comparisons even though they pattern with the dissertation results. Gemma3:1b's failure to reproduce the effect on abstracts is unexplained and sits uncomfortably with its Study 1 behaviour. And the paper is still under peer review, with only Study 2 preregistered (aspredicted.org/bb3b4t.pdf), so the headline contrast between human and model bias should be treated as a strong but not yet vetted result.

Connected Concepts

• Automated Assessment — the practice the paper evaluates and the site of the fairness risk it documents
• AI Detection — undermined by the same entanglement between authorship cues and quality judgement
• Bias Mitigation — the corrective the author demands: bias auditing and validation before deployment
• Academic Integrity — the integrity stakes when AI-assisted work is systematically advantaged
• Assessment Validity — the validity question of whether model grades measure knowledge or LLM conformity
• Automated Essay Scoring — the specific system class the self-preference finding most directly threatens
• Evaluative Judgment — human and model judgement compared on the same rating task and scale
• Equity — unequal tool access converted into unequal grades
• Generative AI — the technology whose output the evaluators were shown to favour
• Large Language Models (LLMs) — the three model families tested as graders across a wide capability range
• Higher Education — the psychology dissertation context from which the corpora were drawn
• Quantitative Research — the experimental and regression-based method behind every reported effect

Connected Articles

• LLMs Do Not Grade Essays Like Humans — Evidence that LLM grading diverges from human essay judgement
• Comparing GPT and human raters in essay assessment: Variability, bias, and the potential of LLM-based scoring — Agreement between GPT and human raters on essay assessment
• Ethical implications of prompt injection in AI-mediated grading: An adversarial red-team evaluation — A second failure mode where submissions manipulate their own grader
• Know When to Trust: Making AI Scoring More Reliable for Educational Assessment — Reliability of AI scoring and when it can be trusted
• A Framework for Evaluation of Large Language Models in Essay Assessment: Reliability, Alignment, and Causal Reasoning — Framework-level treatment of LLM essay assessment reliability
• Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models — Benchmarking GenAI graders against expert human markers
• Validity of Large Language Model Comparative Judgment for Universal Writing Screening — Comparative judgement as an alternative writing-screening approach
• Heads We Win, Tails You Lose: AI Detectors in Education — What AI detectors can and cannot establish about authorship
• Who wrote this? Evaluating the reliability of AI detection tools in higher education — Reliability limits of AI-generated-text detection
• Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content — Using LLMs to identify LLM-written educational text

Citation

Mak, M. (2026). Risks of Using Large Language Models in Grading: LLMs and Humans Prefer LLM-Generated Writing Over Human's but LLMs Show a Stronger Systematic Bias. PsyArXiv Preprints (under peer review).