A systematic comparison of Large Language Models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor

Synthesis: Jukiewicz (2026) presents the first large-scale, side-by-side comparison of 18 contemporary Large Language Models for Automated Grading of programming assignments, grading 6,081 complete question–answer pairs collected across four years of an introductory Python course for cognitive science students (~6,500 raw records, empty submissions removed). The models spanned four vendors — Anthropic (claude-3-5-haiku, claude-sonnet-4, claude-opus-4-1), DeepSeek (deepseek-chat, deepseek-reasoner), Google (gemini-2.0-flash-lite/flash, gemini-2.5-flash-lite/flash/pro), and OpenAI (gpt-4.1-nano/mini, gpt-4.1, gpt-4o-mini, gpt-4o, gpt-5-nano/mini, gpt-5) — queried via a standardized Chain-of-Thought prompt that requires each model to solve the problem itself, compare with the student submission, and return a grade of 0 (incorrect), 0.5 (almost correct), or 1 (correct) under default API parameters. Statistical analyses — grade distributions, means and standard deviations, Spearman rank correlations, Cohen's κ, Conover post-hoc tests with Holm/Holm–Šidák correction, and clustering — revealed systematic differences in grading strictness (mean scores from 0.290 for gpt-4.1-nano to 0.608 for claude-haiku-3.5), with models from the same vendor clustering together (e.g., gpt-5-family ρ ≈ 0.80–0.89, Gemini family ρ ≈ 0.79–0.83). Full-scale flagship models consistently outperformed their smaller "mini"/"nano" counterparts, yet even the best model achieved only moderate agreement with human teacher grades. The author concludes that model choice is not neutral and argues for careful selection, transparent metrics, and a human-in-the-loop approach.

Key Findings

Large-scale comparison. Eighteen Large Language Models (LLMs) from four vendors were compared on 6,081 graded student submissions, the first such large-scale side-by-side Assessment of contemporary models, including the GPT-5 family and Claude Opus 4.1 released in 2025.

Grading-style divergence. Models split along a lenient–restrictive continuum: gpt-4o, gpt-4o-mini, and claude-haiku-3.5 awarded the most maximum (1.0) scores (lenient), while deepseek-reasoner and gpt-4.1-nano were dominated by failing (0.0) grades (restrictive); claude-sonnet-4 and gemini-2.0-flash-lite used the intermediate (0.5) category heavily. Mean grades ranged from 0.290 (gpt-4.1-nano) to 0.608 (claude-haiku-3.5).

Vendor clustering. Spearman correlations (≈0.55–0.89) were highest within vendor families — gpt-5-mini/gpt-5/gpt-5-nano (ρ ≈ 0.80–0.89), the Gemini family (ρ ≈ 0.79–0.83), and gpt-4.1 vs. gpt-4.1-mini (ρ ≈ 0.85) — suggesting shared algorithmic approaches. Models deviating most in style (claude-haiku-3.5, gpt-4.1-nano) correlated only 0.55–0.68 with others.

Agreement is weaker than correlation. Cohen's κ ranged widely (≈0.20–0.74), always below the corresponding Spearman coefficients: within-family κ peaked around 0.74 (gpt-5 vs. gpt-5-mini) and 0.68 (gpt-4.1 vs. gpt-4.1-mini), while deviant models rarely exceeded κ ≈ 0.30. Models thus share overall grading tendencies but diverge on point-by-point scores.

Scale matters. Full-scale flagship models consistently outperformed their smaller "mini" and "nano" counterparts in alignment with human grades.

Limited human alignment. Despite strong internal agreement among models, alignment with the human teacher's grades remained limited — even the best-performing model achieved only moderate reliability, underscoring the need for human oversight in CS Education deployment.

Implication. The choice of LLM architecture and vendor materially affects grading outcomes in CS Education, a key consideration for Automated Grading and Assessment design; the author recommends careful model selection, transparent reporting, and a human-in-the-loop approach.

What this means for practice

• Software developers. Pin the exact model version and default parameters in any grading pipeline you ship: this study held one Chain-of-Thought prompt fixed while mean grades ranged from 0.290 (gpt-4.1-nano) to 0.608 (claude-haiku-3.5), so a quiet vendor upgrade changes scores and must trigger re-validation.
• Software developers. Do not swap in a "mini" or "nano" model to save cost without measuring agreement against human grades: every full-scale flagship (gpt-4.1, gpt-5, gemini-2.5-pro) outscored its smaller sibling in ICC, and only claude-haiku-3.5, a lightweight model, matched the best agreement of all 18.
• Instructors. Treat any single model's output as a provisional signal rather than the grade: the strongest model–teacher agreement in the study was ICC = 0.470 (claude-haiku-3.5), below the 0.75 threshold normally read as good reliability, while the teacher's own mean grade was the highest of any source compared.
• Instructors. Do not read agreement among the models you happen to have access to as independent confirmation: the Claude, GPT, Gemini, and DeepSeek families clustered by vendor (gpt-5 family ρ ≈ 0.80–0.89; κ ≈ 0.74 for gpt-5 vs. gpt-5-mini), and the author warns that ICCs above 0.8 against a majority-vote consensus may only show consistent replication of the same errors.
• Instructors. Keep a human in the loop and publish your metrics: because model choice is not neutral and even the best model reached only moderate reliability, record which model graded which task and against what rubric so a student can contest a decision.

Limitations

• The human reference grades came from one teacher per topic, assigned under normal classroom conditions and explicitly not blind — the teacher knew the student's identity — so the comparison standard is a single unvalidated rater.
• All 6,081 graded question–answer pairs came from one four-year introductory Python course for cognitive science students; the design includes no second course, level, language, or assignment format, so transfer beyond that context is untested.
• Every model was queried with the same Chain-of-Thought prompt under default generation parameters in August 2025, with temperature, decoding, and safety filters left unmodified, making the findings a version- and prompt-specific snapshot.
• The author states that low human–model agreement may also reflect a prompt that failed to communicate all nuances of pedagogical grading, and the prompt itself was never varied or ablated.

Citation

Jukiewicz, M. (2026). A systematic comparison of Large Language Models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor. Computers and Education Open, 100364. https://doi.org/10.1016/j.caeo.2026.100364