From evaluation to emulation: LLMs as agents of iterative pedagogical design

Synthesis: From evaluation to emulation: LLMs as agents of iterative pedagogical design — Yaşar, Kashyrskyy, Xie, and Bulseco (2026) reframe large language models from static graders into emulators of pedagogical reasoning, showing that rubric-guided prompting and role-aware feedback simulations let GPT-4 approximate human evaluative judgment in design-based learning. Using Situated Learning theory, iterative design pedagogy, and a cognitive framework for scientific and engineering thinking, the authors evaluated 80 student design posters across instructor, peer-reviewer, and grant-reviewer roles, finding that iterative rubric co-refinement raised LLM–human agreement from 54.75% to 81.25% and produced role-sensitive feedback variation. The study positions the rubric as a mediating interface between human pedagogical intent and machine inference, advancing AI in education from automation toward pedagogical emulation.

Key Findings

• Rubric engineering drives LLM–human convergence. Initial LLM–human agreement was poor (Cronbach's Alpha = 0.393; Kappa −0.06 to 0.18), but after iterative rubric refinement — clarifying performance descriptors and explicitly accepting implicit indicators of learning — mean agreement rose from 54.75% to 81.25%, with the largest gains in the cognitively demanding Iteration & Reflection category. Final rubric-tuned LLM ratings reached Alpha = 0.798 and Kappa 0.40–0.55, evidencing convergence toward shared evaluative reasoning.

• Rubrics as semantic interfaces for AI. The study treats assessment criteria as revisable design artifacts rather than fixed instruments, positioning the rubric as a mediating interface between human pedagogical intent and machine inference. Rubrics engineered for LLMs must balance precision and flexibility — too vague invites free interpretation, too rigid reduces the model to pattern-matching.

• Role-aware prompting yields distinct evaluative feedback. The same artifact evaluated under instructor, peer-reviewer, and grant-reviewer prompts produced qualitatively different tone and focus — instructors were encouraging and process-oriented, peers supportive and conversational, grant reviewers formal and outcomes-oriented. These differences were epistemic, not merely stylistic, foregrounding different aspects of design practice.

• Structural alignment in clustering. K-means clustering of human and LLM score matrices showed highly correlated cluster centroids (r = 0.89), with 85% of posters classified into the same or adjacent performance clusters, indicating LLMs can reproduce latent structure in student work when scaffolded with a semantically precise rubric.

• LLMs as calibration and co-design partners. Beyond scoring, LLMs served as rubric stress-testing and semantic-debugging tools, and post-revision demonstrated greater consistency than some human raters in applying performance thresholds — useful for norming sessions and formative peer feedback environments.

• Human-in-the-loop oversight remains essential. The authors caution that LLMs can misinterpret nuance, hallucinate rationale, or project false confidence; role fidelity depends heavily on prompt specificity, and models occasionally blend roles. They advocate for human-in-the-loop assessment where educators review and refine LLM outputs rather than treat them as authoritative.

Connected Concepts

• Large Language Models (LLMs)
• Feedback
• Formative Assessment
• Learning Design
• Design-Based Research
• Situated Learning
• Prompt Engineering
• Training Pedagogical LLMs for Tutoring
• AI Feedback Quality
• Human-in-the-Loop

Connected Articles

• LLM-generated formative feedback in education: A qualitative systematic literature review — LLM-generated formative feedback
• Comparing GPT and human raters in essay assessment: Variability, bias, and the potential of LLM-based scoring — LLM vs. human raters in essay assessment
• The care-full craft of feedback in an age of generative AI — Care-full feedback with generative AI
• The Effect of High-Frequency, Automatically-marked Formative Assessments on Student Outcomes in A-Level Sciences — Automated formative assessment in science
• Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior — LLM tutoring in exploratory learning
• Pre-service teachers' agency during their interactions with generative AI while designing for learning - a process view — Generative AI in design-based learning

Citation

Yaşar, O., Kashyrskyy, A., Xie, C., & Bulseco, D. (2026). From evaluation to emulation: LLMs as agents of iterative pedagogical design. International Journal of Artificial Intelligence in Education, 36, 100013.