A Decade of Reflection and Thematic Review on Artificial Intelligence's Impact on Educational Measurement

Synthesis: In brief: Xiong and Li map a decade (2015–present) of AI's impact on educational measurement through a thematic review that integrates five major international conference trends with peer-reviewed literature. They identify three evolutionary eras — the Formative Era (2015–2018), the Expansion Era (2019–2022), and the ongoing Generative Era (2023–present) — and organize AI's progression from an operational tool to a co-designer of assessment and learning through an Efficiency–Enhancement–Transformation framework, spanning four themes: AI on scoring and item generation, psychometric modeling, assessment innovation and process data, and fairness/ethics/equity.

This review traces how educational measurement has undergone a "computational turn" over the past decade, moving from its historical grounding in Classical Test Theory (CTT) and Item Response Theory (IRT) toward AI-integrated methods. The authors analyze conference trends (NCME, AERA, IAFOR, ASCILITE, and others) alongside the peer-reviewed corpus to quantify the rising proportion of AI-related measurement papers and to assign topics via Latent Dirichlet Allocation topic modeling. Their Efficiency–Enhancement–Transformation framework captures AI's evolving role: from efficiency gains in scoring and item generation, to enhancement of psychometric modeling, to transformation of assessment through process data and new paradigms that integrate measurement theory with AI methods.

The review surfaces both opportunities and challenges. Opportunities include scalability and richer diagnostic inference; challenges include interpretability, algorithmic bias, and construct validity. A key forward-looking conclusion is the need to reconceptualize educational constructs in the context of human–AI interaction and to adopt AI in Educational Measurement responsibly, with emphasis on transparency and equity.

Key Findings

• Three evolutionary eras: Formative (2015–2018), Expansion (2019–2022), and Generative (2023–present).
• Efficiency–Enhancement–Transformation framework organizes AI's progression from an operational tool to a co-designer of assessment and learning.
• Four key themes: (a) AI on scoring and item generation, (b) AI on psychometric modeling, (c) AI on assessment innovation and process data, and (d) AI on fairness, Ethics, and equity.
• Opportunities: scalability and richer diagnostic inference from AI-integrated measurement.
• Challenges: interpretability, algorithmic bias, and construct validity.
• Argues for a new paradigm integrating measurement theory with AI methods and for reconceptualizing constructs in the context of human–AI interaction.

What this means for practice

• Assessment professionals. In high-stakes decisions, favor interpretable models over the highest-performing ones; the review names the interpretability–performance trade-off as the central challenge of AI in measurement.
• Assessment professionals. Constrain LLM scoring and feedback with rubrics and human validation — the reviewed literature shows LLMs can produce contradictory or inconsistent feedback across different aspects of a single response.
• Assessment professionals. Rewrite constructs and validity arguments for human–AI collaboration rather than unassisted human performance; the review argues Classical Test Theory and IRT validity models built on solo work no longer fit when AI is available to every learner.
• Researchers. Log and report process evidence such as interaction traces and revision behaviors as part of the validity argument, treating it as evidence of learning-in-action rather than an incidental byproduct.
• Researchers. Reconceptualize fairness and validity as properties of the whole instructional-assessment system, so that measured outcomes reflect equitable learning opportunities rather than differential access to AI tools.

Limitations

• The journal synthesis rests on a final corpus of 313 peer-reviewed English articles published 2015–2025 and retrieved from four databases (Scopus, Web of Science, ERIC, PsycINFO); preprints, editorials, and non-English work were excluded by design.
• Studies focused purely on algorithmic development without a demonstrated educational application, and AI-for-teaching studies without any measurement or assessment component, were excluded, so all four themes describe a deliberately measurement-facing slice of the literature.
• Conference evidence comes from five venues (EARLI, NCME, AERA, ACE, ASCILITE), and the thematic structure comes from a four-topic LDA solution chosen by perplexity and coherence scores plus expert judgment — choices that shape the reported eras and themes.
• It is a structured thematic synthesis, not a quantitative meta-analysis: no pooled effect sizes are reported, so the review cannot estimate how much AI improves measurement outcomes.

Citation

Xiong, J., & Li, F. (2026). A decade of reflection and thematic review on artificial intelligence's impact on educational measurement. Educational Research Review, 51, 100789.