Teachers' Collaborative Evaluation of AI-Generated Content: Insights from a Professional Development Workshop

Synthesis: This study investigates how middle-school science teachers evaluate AI-generated assessment questions and how they perceive the role of collaborative evaluation in professional development. Sixty science teachers reviewed open-ended questions on lower- and higher-order thinking skills produced by ChatGPT, rating them through their disciplinary, pedagogical, and curricular judgment. The mixed-methods findings show that collaborative evaluation helps teachers apply conceptual-precision judgment to AI content and surface risks such as reinforcing Misconceptions about AI. The paper underscores the value of professional development that positions teachers as critical evaluators of AI-generated material rather than passive consumers.

Background

As GenAI becomes integrated into education, AI-generated content risks reinforcing misconceptions and propagating imprecise reasoning — particularly in disciplines that demand conceptual precision. Question writing for assessment therefore requires critical evaluation by educators, but little is known about how teachers exercise that judgment collectively.

Key Findings

1. Sixty middle-school science teachers reviewed ChatGPT-produced questions spanning lower-order and higher-order thinking skills.
2. Teachers drew on disciplinary, pedagogical, and curricular judgment to rate AI-generated content, applying conceptual-precision criteria.
3. Collaborative evaluation emerged as valuable professional development, helping teachers surface and address the risk that AI content reinforces misconceptions.
4. A mixed-methods design combined quantitative ratings with qualitative insights on teachers' perceptions.
5. The findings position teachers as critical evaluators of AI material, supporting critical engagement with GenAI in the classroom.

What this means for practice

• Instructors. Have discipline-specific groups review AI-generated questions against stated criteria — scientific accuracy, linguistic clarity, and curriculum alignment — before they reach students. Teachers in this study caught subtle errors in items that read as scientifically correct, such as attributing a swing's behavior on the International Space Station to air resistance rather than microgravity.
• Instructors. Treat AI output as a draft for revision: teachers rated higher-order questions lowest on curriculum alignment (M = 3.47, SD = 0.92) and described them as conceptually ambiguous, so plan to rewrite or replace those items rather than assigning them as generated.
• Instructors. Screen for syntactic accessibility separately from factual accuracy: teachers rated a question 4 out of 5 for clarity while still warning that embedded clauses and abstract phrasing would challenge middle-school readers.
• Faculty developers. Make the evaluation a structured workshop task rather than a demonstration: mixed groups of three to four novice and veteran teachers, working from a disciplinary text and a fixed prompt with a shared rating form, produced specific and shared critiques.
• Faculty developers. Withhold AI tools during the critique phase so judgments rest on disciplinary, pedagogical, and curricular knowledge; this workshop barred external resources and GenAI while teachers evaluated the items.

Limitations

• Sixty middle-school science teachers from eight schools in one district, all taking part in a single 90-minute workshop within one year-long PD seminar; the authors caution that the lower-order versus higher-order differences should be interpreted cautiously given the exploratory sample.
• Each discipline-specific group of three to four teachers evaluated only one lower-order and one higher-order question, so the observed rating pattern rests on a very small stimulus set.
• All items were generated by a single model — the free browser version of ChatGPT-4o — under one standardized prompt, tying the findings to that model's behavior at that moment.
• Ratings and reflections were collected at group level, teachers knew the content was AI-generated, inviting evaluation bias and group dynamics, and no pre/post measure of evaluative skill was taken, so the study documents perceptions rather than demonstrated evaluation ability.

Citation

Gat, I., Usher, M., & Barak, M. (2026). Teachers' Collaborative Evaluation of AI-Generated Content: Insights from a Professional Development Workshop. EdArXiv preprint.