PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback

Synthesis: PsyScore integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation. Three modules: (1) Trait-Adaptive Neural IRT Scorer incorporating GPCM for precise ability estimation (QWK 0.747, besting prior SOTA 0.722); (2) ZPD-Scaffolded Feedback Generator conditioning multi-agent Feedback on diagnosed ability; (3) Multi-Perspective Feedback Evaluation via pairwise preferences and simulated revisions. Demonstrates that psychometrically-aware automated essay scoring (AES) can transform from summative scoring into formative diagnosis.

Key Findings

• Problem: AES systems treat scoring and feedback as separate; neural IRT scoring lacks interpretability; LLM feedback is ability-agnostic.
• Solution: Unified psychometric latent space (θ) via Neural GPCM Trait-Adaptive Scorer.
• Scoring performance: QWK 0.747 (besting prior SOTA 0.722); 1st in 6/8 ASAP++ prompts and 10/11 trait dimensions.
• Feedback quality: ZPD-based strategy mapping (Explicit Correction for θ<-1, Scaffolding for -1<θ<1, Socratic for θ>1). Multi-agent fusion from Llama-4-Scout, Qwen3-235B, GPT-4o.
• Evaluation: >90% win rate in Actionability vs GPT-4o, Llama-4-Scout, Qwen3.
• Implication: psychometrically-aware AI can unify assessment and instruction, transforming automated essay scoring from summative to formative.

What this means for practice

• Instructors. Vary feedback strategy by diagnosed proficiency: give explicit correction below θ = −1, scaffolded hints between −1 and 1, and Socratic questioning above 1, rather than one style for the whole class.
• Instructors. Return trait-level scores (Content, Organization, Conventions, Word Choice, Sentence Fluency, Voice) next to the holistic score so students get direction on the dimension that is actually weak.
• Designers. Build the scoring layer on an interpretable IRT latent-ability estimate instead of an opaque regression head, so score and feedback share one ability representation.
• Researchers. Judge generated feedback with both pairwise preference judgments and simulated student revisions, because surface fluency ratings and actionability diverge.

Limitations

• Scoring results come from the ASAP++ benchmark (8 prompts, single 6:2:2 split with 20% held out), so the reported QWK of 0.747 is a benchmark figure rather than an estimate from live classroom writing.
• Feedback quality was judged on a stratified sample of 400 essays (50 per prompt) and by a double-blind panel of only three senior education experts rating 80 essays.
• The trait-adaptive scorer depends on fine-grained analytic labels (e.g., Voice, Organization) that the authors note are scarce in operational assessment settings, restricting it to corpora with trait annotations.
• The ZPD scaffolding gains (a 17.38% normalized gain for θ < −1) were measured through simulated revisions; the authors state the protocol cannot model motivation, epistemic trust or cognitive fatigue, and no controlled classroom trial was run.

Citation

Xia, W., Wu, J., Shi, H., Wang, X., & Zheng, C. (2026). PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback. East China Normal University / arXiv cs.CL preprint.