From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations

Synthesis: Yan, Xiong, Li & Chen (2026) reposition LLMs from benchmark targets to auxiliary evidence sources for interpreting programming-exam difficulty, showing that AI difficulty estimates correlate strongly with student pass rates across parallel-class finals (rho ≈ −0.87 at problem level) while explicitly bounding that these scales must not be used for individual student evaluation or automatic grade adjustment.

Key Findings

1. AI pass rate tracks student performance. In a synchronous eight-problem final exam where ten models solved alongside 120 students, AI pass rate correlated positively with student pass rate (Spearman rho = 0.866), and a solving-based composite difficulty index correlated negatively with it (rho = −0.905).

2. Strong problem-level calibration across exams. Across 79 problems from 11 parallel-class final exams, AI overall difficulty correlated with problem-level pass rate at rho = −0.871 and with non-attempt rate at rho = 0.800; a 26-problem longitudinal data-structures sample gave −0.829 and 0.883.

3. Boundary condition in introductory courses. A 106-problem CS101 sample marked the limit: problem-level correlation weakened to rho = −0.552 and exam-level correlation across 16 exams was near zero, with cohort composition dominating exam-level outcomes. Exposure-discount and duplicate-problem perturbation tests did not change the direction of findings.

4. Explicit use-and-abstention boundaries. The single-reviewer design, unverifiable model identity, and review-output instability mean AI difficulty scales are suitable for problem validation, parallel-class fairness discussion, and longitudinal quality tracking — but must not drive individual student evaluation or automatic grade adjustment.

Implications

This study reframes the role of LLMs in Assessment from "evaluated object" to "evaluation aid," contributing a methodology that layers AI evidence with student performance, item exposure, and Learning Analytics to interpret exam difficulty. It connects to the growing literature on Psychometrically Aware AI and Confidence Aware AI Assessment, where model outputs are treated as one noisy signal among several rather than as ground truth.

For CS Education and CS Education practice, the finding that AI difficulty correlates with student outcomes at the problem level supports using LLMs to flag mis-calibrated items across parallel sections and to track item quality longitudinally. The clear abstention guidance is the crucial guardrail: cohort composition effects in introductory courses and the fragility of single-reviewer estimates caution against high-stakes automation, aligning with Human In The Loop AI design principles.

The work also illustrates the epistemic limits of Item Response Theory-style difficulty estimation when grounded in model rather than human response data, and reinforces the need for verification and AI Ed Evaluation frameworks that keep AI in a supporting rather than deciding role.