Auto-marking short answer questions in science: The foundational years of transformer-based models from BERT to GPT-4

Synthesis: Auto-marking short answer questions in science: The foundational years of transformer-based models from BERT to GPT-4 — Morley, Walland, and Vidal Rodeiro scoping-review the transformer-based auto-marking of short-answer science questions during the foundational period 2017–early 2024, coding 21 articles under PRISMA-ScR guidelines. BERT models (and variants) dominated, peaking in 2021 before GPT-based approaches arrived, while models augmented with domain-specific data such as textbooks and marking rubrics consistently outperformed those without. The review surfaces enduring threats to reliability, explainability, and fairness and argues auto-markers should support rather than replace human examiners.

Key Findings

• Model landscape and temporal shift. BERT (base and variants such as RoBERTa, DistilBERT, SciBERT) was the most common transformer approach, used in 20 of 21 studies and peaking in 2021; GPT models (GPT-1/2, GPT-3.5, GPT-4) appeared from roughly 2022, adopted via Prompt Engineering rather than fine-tuning, reflecting a field-wide move from fine-tuning smaller models toward prompting larger LLMs.
• Datasets and performance. The SciEntsBank corpus dominated (2-way, 3-way, and 5-way items with unseen-answers/questions/domains tests), alongside ASAP SAS, Beetle, BEAR, TIMMS and others — all US-collected. BERT models generally outperformed earlier auto-markers on SciEntsBank, though no study yet benchmarked GPT models there; one direct comparison found GPT-3.5 outperformed BERT base across items.
• Domain data and data augmentation help. Models incorporating additional domain knowledge — textbooks, marking rubrics, science journals, further pre-training, or meta-learning — consistently outperformed models without it; GPT-generated synthetic data also improved fine-tuned BERT performance, while rubric-aware and chain-of-thought prompting lifted GPT accuracy.
• Reliability and validity concerns. Auto-markers can learn "spurious correlations" with surface features (punctuation, grammar, wording) rather than construct-relevant scientific understanding, and small input changes can produce large output differences, threatening reliability and construct validity.
• Explainability and bias gaps. Few models could justify marks in human-comprehensible terms, GPT "rationale" generation and chain-of-thought prompting only partially address this, and bias across demographic and linguistic groups was rarely examined — undermining Trust and raising ethical stakes, especially in high-stakes settings.
• Recommendations for the field. The authors call for more diverse and shared datasets, comprehensive evaluation frameworks (reliability, validity, fairness, explainability, robustness, practical utility), transparent and hybrid human-machine scoring models, rigorous bias analyses, and movement beyond simple classification toward multi-mark-point and partial-credit scenarios.

What this means for practice

• Assessment professionals. Augment auto-markers with domain-specific data — textbooks, marking rubrics, subject journals, or further pre-training — since every reviewed study that added domain knowledge outperformed models without it.
• Assessment professionals. Stress-test a marker against trivial surface edits (wording, punctuation, grammar) before trusting a score: models can learn spurious correlations with surface features instead of scientific understanding, and small input changes can produce large output differences.
• Instructors. Keep a human examiner in the loop and use auto-marking to support rather than replace professional judgment, above all in high-stakes settings where performance drops on unseen questions or domains.
• Researchers. Report fairness by demographic and linguistic subgroup and require models to justify marks in educator-readable terms; the review found that bias analysis was rarely examined and that no reviewed approach reached the standard needed for educational accountability.
• Assessment professionals. Do not assume GPT-era results transfer to established corpora: no reviewed study had yet benchmarked GPT models on SciEntsBank, the dataset used by 10 of the 21 articles.

Limitations

• The review rests on 21 articles drawn from a single database (Scopus) after a search that started with 252 records; the authors describe it as a scoping review with no research protocol and state that it did not comprehensively cover the available literature.
• One researcher with data-science expertise coded all 21 articles in depth (MAXQDA 2024), and no article was removed for quality, so the charting and quality judgment rest on a single analyst.
• Scope was deliberately narrow — English-language natural-language responses on science content (biology, chemistry, physics) tested with transformer-based technology — so other subjects, languages, and non-transformer approaches were excluded, and many models were evaluated on US-collected corpora.
• The publication window ran from 2017 to 18 March 2024, when the authors note the 2024 data were still incomplete, so the GPT-model picture is an early snapshot.

Citation

Morley, F., Walland, E., & Vidal Rodeiro, C. (2026). Auto-marking short answer questions in science: The foundational years of transformer-based models from BERT to GPT-4. International Journal of Artificial Intelligence in Education, 36, Article 100005.