Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence

Synthesis: Takahashi and colleagues automate hierarchical diagnostic reasoning (HDR), an applied-competence task in which learners find and explain deliberately embedded errors in a short case. Using GPT-4o and prompt design alone, with no fine-tuning, the team generated new problems, scored descriptive answers, and produced structured Feedback. Model-written problems matched human-authored ones on internal consistency (Cronbach's α = 0.78) and difficulty, and human–model scoring agreement reached 98%–100% once the prompt carried worked examples from human grading. Model feedback matched instructors on persuasiveness and usefulness, though usefulness ran lower.

Key Findings

1. Generated problems reproduced human difficulty. GPT-4o-created and original HDR problems both yielded Cronbach's α = 0.78, and Fisher's exact tests found no significant difference in their score distributions across 100 participants.
2. HDR measured applied judgment, not reading. Correlation with the reading-comprehension control was low (α = 0.36 and 0.41) against 0.78 between the two HDR problems.
3. Scoring accuracy hinged on worked examples in the prompt. Without hints, Q2 precision fell to 0.42; with human-scored correct and incorrect examples added, agreement reached 98% or higher on every question.
4. Two of three questions reached perfect agreement. Under the hint condition Q1 and Q2 both hit 100% accuracy, and GPT scoring its own generated problem gave Fleiss' κ of 1.00 for both.
5. Human grading carried a measurable cost. Five expert graders spent 138 minutes scoring 117 responses plus 30 minutes of consensus, about 1 minute and 30 seconds per learner, against near-instant model scoring.
6. Feedback matched instructors on both rated dimensions. Human and ChatGPT feedback scored 3.67–4.05 for persuasiveness and 2.95–3.54 for usefulness, with no significant difference on the Wilcoxon signed-rank test.
7. Feedback creation dominated the human workload. Preparing, drafting, and re-checking feedback for 117 questions took 860 minutes, and the full human loop reached about 11 minutes and 30 seconds per learner.

The task structure that makes automation possible

HDR asks learners to read a scenario, identify the embedded error, and explain why it is wrong. The original problem is a seven-paragraph Segmentation-Targeting-Positioning case about an electronics manufacturer carrying three embedded errors. Each of the three questions scores one point for the multiple-choice selection and one for the description, for a 0–6 total. Because answers are anchored to a predefined frame, learner responses converge and grading becomes reproducible. The authors argue that pairing a higher-order thinking demand with a constrained response format gives HDR its affinity with a general-purpose Large Language Models (LLMs): it yields structured prose about identifiable Misconceptions about AI, exactly the material a language model handles well.

Prompt design, not retraining, decided scoring quality

Two conditions were compared. Without hints the prompt carried only the scoring criteria; with hints it also carried multiple human-scored correct and incorrect examples. Each answer was scored nine times and the final label taken by majority vote. The failure mode without hints was instability rather than ignorance: a Q2 answer about weight-loss users was over-interpreted as correct when humans judged it incorrect, while a Q1 answer mentioning user attributes was judged too strictly. Adding examples aligned model judgments with expert consensus and raised Q1 accuracy to 1.00. For teams doing Prompt Engineering, criteria alone under-determine the boundary of a correct answer; boundary cases have to be shown. The prompts also output a rationale, which the authors treat as serving transparency.

Feedback that persuaded, and where it was less useful

Feedback was standardized to address the reason for an error and the learner's good points, with an MBA instructor's version for comparison. Seventy-four of 80 respondents with no missing data rated both variants on a 5-point Likert scale, blinded to source and with order randomized. Persuasiveness averaged 3.67–4.05 and usefulness 2.95–3.54 for both sources, with no significant difference. Since both measures are self-reports, they show perceived quality, not learning. The gap is informative: some respondents said the model's feedback used technical terms they could not apply, which the authors attribute partly to the sample being working adults rather than MBA students.

Why the authors call this a redesign, not a speedup

The paper places the work at the Modification stage of the SAMR Model, arguing that automating generation, scoring, and feedback reconstructs the assessment activity rather than merely digitizing it. The strongest claim is about repetition: reusing one case invites memory and answer reuse, while a model can produce structurally equivalent but contextually different cases, which the authors say suppresses item bias in re-measurement. They see the format extending to STEM, nursing and medical, and legal education settings. The counterweight: because LLM evaluation rewards logic and structure, students may drift toward model-friendly answers at the expense of depth, so the authors call for hybrid AI–human evaluation rather than replacement.

What this means for practice

• Instructors. Give the scoring prompt worked examples from your own prior grading, not just a rubric; criteria alone left the model over-penalizing answers human experts scored consistently.
• Assessment designers. Build the exemplar bank as a versioned artifact: the gap between 0.42 Q2 precision without examples and 1.00 with them is your design margin.
• Administrators. Price automation against the measured human cost: 138 minutes of expert scoring plus 30 minutes of consensus for 117 responses, and 860 minutes for feedback preparation, writing, and re-checking.
• Researchers. Treat perceived quality and learning as separate outcomes; persuasiveness and usefulness ratings say nothing about whether feedback changes later performance.

Limitations

• Scoring accuracy depends on prompt design; the authors state that its robustness to changes in context and task characteristics still has to be verified.
• Only STP analysis was tested, so expansion to other frameworks still required evaluation despite the cross-disciplinary argument.
• The study used a single model, OpenAI's GPT-4o, which the authors note may behave differently from other LLMs; robustness across models was left to future work.
• Feedback effects were not followed up: whether model feedback changes understanding or behavior needs a long-term, qualitative study, and the feedback experiment analyzed 74 of 80 respondents.

Connected Concepts

• Automated Assessment
• Assessment Validity
• Formative Assessment
• Feedback
• Prompt Engineering
• Large Language Models (LLMs)
• Automated Question Generation
• Critical Thinking
• SAMR Model
• Human-in-the-Loop
• Authentic Assessment
• Educational Measurement
• AI Feedback Quality
• Evaluative Judgment
• Explainable AI

Connected Articles

• CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback — CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback
• AI-assisted, instructor-supervised grading and feedback in higher education: Design and evaluation of an end-to-end pipeline — AI-assisted, instructor-supervised grading and feedback in higher education: Design and evaluation of an end-to-end pipeline
• Auto-marking short answer questions in science: The foundational years of transformer-based models from BERT to GPT-4 — Auto-marking short answer questions in science: The foundational years of transformer-based models from BERT to GPT-4
• Agreement and error in automated scoring of student marketing posts — Agreement and error in automated scoring of student marketing posts
• Can AI Evaluate Assessment? A Study of Large Language Model Meta-Assessment Performance — Can AI Evaluate Assessment? A Study of Large Language Model Meta-Assessment Performance
• AI-mediated authentic assessment and metacognitive reflection: A mixed-methods study of the AAIWA model — AI-mediated authentic assessment and metacognitive reflection: A mixed-methods study of the AAIWA model
• Beyond Detection: Redesigning Authentic Assessment in an AI-Mediated World — Beyond Detection: Redesigning Authentic Assessment in an AI-Mediated World
• Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study — Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study
• Mapping the Integration of AI into Business Education: Insights from a Decade of Research — Mapping the Integration of AI into Business Education: Insights from a Decade of Research
• Generating a Student-Informed Teaching and Learning Conceptual Framework for GenAI in Business Schools: A Case Study — Generating a Student-Informed Teaching and Learning Conceptual Framework for GenAI in Business Schools: A Case Study

Citation

Takahashi, S., Yoshikawa, A., Kose, M., Suzuki, K., Inoue, C., Watanabe, Y., & Sawada, M. (2026). Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence. arXiv:2609.25790.