On this page

Synthesis: Xu and colleagues (2026) reviewed 60 empirical studies published between 2015 and 2025 on AI in teaching evaluation, then narrowed to 19 that actually reported validation evidence or demonstrated educational impact. The result is a field with a lot of models and very little measurement work: reliability and validity evidence is thin and inconsistently reported, bias is discussed more often than it is tested, and the human–AI division of labor is usually asserted rather than designed. The review's contribution is to make the evidentiary gap explicit enough to be a research agenda.

Key Findings

  • The literature is large and shallow. 60 eligible empirical studies since 2015; only 19 formed the core evidence base because they reported reliability or validity evidence, or demonstrated educational impact.
  • Validation is the missing layer. Studies report model performance — accuracy, agreement with human raters — rather than the psychometric evidence an institution needs before using a system for personnel decisions.
  • Bias is named more than measured. Fairness concerns appear across the corpus, but few studies test for differential performance by instructor gender, discipline, seniority or language.
  • Human–AI roles are asserted, not designed. Most systems position the model as a scorer or summariser with the human as reviewer, and few studies test whether that arrangement changes the judgment.
  • Institutional use is outpacing the evidence. The review's framing is that teaching evaluation is a governance function, so an unvalidated model entering it carries consequences for careers.

What the review covers

The search covered teaching, student, course, faculty and teacher evaluation crossed with artificial intelligence, machine learning, deep learning, natural language processing and large language models, then crossed again with evaluation-quality dimensions: bias, reliability, validity and fairness. Searches ran to January 2026 with citation snowballing to July 2026, and eligibility was restricted to English peer-reviewed journal articles, conference proceedings and book chapters from 2015 onward.

The stratification into a 19-study core is the review's most useful move. It separates the studies that could support a decision from the studies that demonstrate a technique, which is exactly the distinction an institution needs when a vendor presents a model.

Why teaching evaluation is the hard case

Student evaluation of teaching is already a contested instrument: response rates are low, scores correlate with grades and with instructor gender, and institutions use them for promotion anyway. Adding a model to that pipeline does not create a measurement problem — it inherits one and hides it behind a number.

The review's framing of human–AI roles matters here. If a model drafts a summary of free-text comments, the human reads a summary the model chose to write, and the judgment is already shaped. Almost none of the 19 core studies test that arrangement against a control, which means the field cannot yet say whether the human in the loop is a check or a rubber stamp.

What this means for practice

  • Administrators: require validation evidence before an AI evaluation tool informs personnel decisions — reliability across raters and occasions, and differential performance by instructor group.
  • Assessment professionals: treat agreement with human raters as a starting point, not as validity. Two raters can agree and both be biased.
  • Institutions: publish the human–AI division of labor. Who sees the model output, at what stage, and what they can override.
  • Researchers: the 41 adjacent studies are the obvious place to add validation work rather than new models.

Limitations

A review is bounded by its search: English-language, peer-reviewed, 2015 onward, with three concept blocks that may miss work framing the same systems as learning analytics or course evaluation. The 19-study core is a judgment about relevance, and the authors do not report a risk-of-bias assessment per study in the abstracted findings. Publication bias toward working systems is likely. The review is a journal pre-proof, so pagination and final copy may change.

Citation

Xu, X., Majid, F., Lai, E., Zhang, S., Fukui, N., & Ravishankar, J. (2026). Artificial Intelligence in Teaching Evaluation: A Systematic Review of Validation Evidence, Bias and Human–AI Roles. Computers and Education: Artificial Intelligence, 100695.

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.