Pei-Yu Tseng, Mahir Akgun, Peng Liu (2026) — arXiv:2606.23629 (cs.HC) 📄 Full text (arXiv)
Tseng et al. (2026) investigate human-machine alignment in LLM-based pretest question evaluation — a critical bottleneck for scalable AI-assisted assessment. Their AI-assisted workflow combines automated generation, rubric-based evaluation, and iterative selection. Through a 2×2 experimental design varying rubric operationalization and evaluation mode, they find that human-machine disagreements are systematic rather than random, rubric revision has a larger effect on alignment than rationale-first evaluation, and the two interventions are complementary. The core insight is that scalable AI-assisted pretesting depends not only on generation capability but crucially on how pedagogical quality is operationalized for machine interpretation. This work contributes to ai-ed-evaluation by providing empirical evidence for aligning llm judgment with human pedagogical standards in formative-assessment contexts, and has direct implications for automated-grading and assessment system design.
Related Pages
- machines-misread-pedagogical-quality -- This page