Wei Xia, Jin Wu, Haoran Shi, Xiangyu Wang, Chanjin Zheng (2026). East China Normal University / arXiv cs.CL preprint ๐ Full text (arXiv)
Overview
PsyScore integrates diagnostic assessment with instructional scaffolding through a shared latent ability representation. Three modules: (1) Trait-Adaptive Neural IRT Scorer incorporating GPCM for precise ability estimation (QWK 0.747, besting prior SOTA 0.722); (2) ZPD-Scaffolded Feedback Generator conditioning multi-agent feedback on diagnosed ability; (3) Multi-Perspective Feedback Evaluation via pairwise preferences and simulated revisions. Demonstrates that psychometrically-aware AES can transform from summative scoring into formative diagnosis.
Key Contributions
Paper 1: Code as Anchor, Memory and Metaphor as Support: Learner Experiences with Multi-View Visualizations
- Problem: Students often ignore well-designed program visualizations; existing cognitive design principles don't explain learner engagement/disengagement.
- Method: Within-subjects study (N=19 undergraduates, post-CS1/CS2) using think-aloud, interviews, and webcam gaze tracking with a multi-representational probe.
- Gaze finding: ~47% of time on code despite visual scaffolds; students without prior experience anchored more in code and ignored metaphor views.
- Three engagement themes: Agency (control over cognitive effort), Representational Fit (wide individual variation), Legitimacy (metaphors seen as childish at university level).
- Implication: multi-representational-tools need attention to affective and social factors, not just cognitive design.
Paper 2: Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring
- Problem: Static-prompt LLM tutoring systems fail to adapt across diverse academic disciplines.
- Solution: Subject-aware prompt routing using 14 pedagogical features extracted from transcripts; contextual bandit formulation with 20 pedagogical prompts.
- Simulation: Router achieves 0.694 vs 0.647/0.64 static baselines (p<0.001).
- Real-world A/B test: N=656 conversations, 359 Dutch high-school students. Stochastic router achieves 28.1% exercise conversion rate vs 19.6% baseline.
- Implication: adaptive-prompt-routing with stochastic sampling improves both efficiency and engagement in real-world tutoring.
Paper 3: Confidence-Aware Automated Assessment of Student-Drawn Scientific Models
- Problem: Automated scoring of student-drawn scientific models lacks reliability indicators, leaving teachers unable to decide when to trust scores.
- Method: Vision Transformer (ViT) with LoRA + confidence-aware framework using test-time perturbations.
- Dataset: Six NGSS-aligned middle school assessment items (477-816 responses each, scored Beginning/Developing/Proficient).
- Key innovation: Response-level confidence enables selective automation โ high-confidence auto-scored, uncertain cases deferred for human review.
- Implication: confidence-aware-ai-assessment enables practical triage between automation and human oversight in educational assessment.
Paper 4: PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback
- Problem: AES systems treat scoring and feedback as separate; neural scoring lacks interpretability; LLM feedback is ability-agnostic.
- Solution: Unified psychometric latent space (ฮธ) via Neural GPCM Trait-Adaptive Scorer.
- Scoring performance: QWK 0.747 (besting prior SOTA 0.722); 1st in 6/8 ASAP++ prompts and 10/11 trait dimensions.
- Feedback quality: ZPD-based strategy mapping (Explicit Correction for ฮธ<-1, Scaffolding for -1<ฮธ<1, Socratic for ฮธ>1). Multi-agent fusion from Llama-4-Scout, Qwen3-235B, GPT-4o.
- Evaluation: >90% win rate in Actionability vs GPT-4o, Llama-4-Scout, Qwen3.
- Implication: psychometrically-aware-ai can unify assessment and instruction, transforming automated essay scoring from summative to formative.
Related Pages
Citation
APA: Wei Xia, Jin Wu, Haoran Shi, Xiangyu Wang, Chanjin Zheng (2026). PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback. arXiv:2606.20287. East China Normal University / arXiv cs.CL preprint.