Bayesian Consensus Calibration of Continuously Evolving IRT Item Banks

Synthesis: Jewsbury and colleagues at Duolingo address an operational problem created by AI-based item generation: when a language-model-driven pipeline supplies new items and predicts their parameters from content, the bank grows continuously, and refitting the entire accumulated response history at every update becomes both expensive and, at contemporary bank sizes, potentially impossible to hold in memory. Their proposal, consensus calibration, calibrates each time period independently and reconstructs the posterior that a single pooled analysis would have produced, using a divide-and-conquer IRT aggregation with two distinguishing features — each period is linked to a common metric by a robust characteristic-curve criterion solved separately for every posterior draw, and the population prior contributed by each period is subtracted out and replaced by a single prior obtained by consensus across periods. Evaluated against a pooled single-run Benchmark on operational Duolingo English Test data spanning four quarterly calibration periods, the consensus aggregate reproduced pooled posterior means almost exactly (r = .998 and .991) and posterior dispersions closely (r = .970 and .920), while an exposure-stratified diagnostic confirmed the prior-de-duplication correction works in the direction intended. The contribution is a computational-psychometric scaling property rather than a new measurement claim: an update costs what the new period costs, not what the whole history costs.

The Calibration Problem That AI Item Generation Creates

Automatic item generation using large language models has produced item banks far larger than manual authoring allowed, and NLP models now predict item difficulty and related parameters from item content — using engineered text features, pretrained-language-model embeddings, or automated machine learning over both — so that items can be operationalized with little or no pretesting. These methods are deployed, not prospective: in operational assessments, feature-based parameters let newly generated items enter scoring before substantial response data accrue. The Duolingo English Test is the paper's exemplar — a high-volume, continuously evolving assessment that combines automatic generation, feature-based parameterization and frequent recalibration.

Hierarchical Bayesian IRT is the natural calibration framework for such a bank. Item parameters are drawn from a population distribution whose hyperparameters are estimated jointly with the items, and explanatory IRT generalizes that population model to a regression of item parameters on item features — the same regression that feature-based prediction estimates. Sparsely observed items are shrunk toward their feature-based predictions, and the posterior supplies the parameter uncertainty on which adaptive selection and scoring depend. The operational difficulty is recalibration: standard practice refits a single model to all accumulated data at each update, so cost grows continuously, and at contemporary bank sizes holding the full history in memory may exceed available resources. The target the authors set is exact and deliberately modest — reproduce what a single fit to all accumulated data would yield, without refitting past data as the bank grows.

The method is positioned against two adjacent literatures. Consensus Monte Carlo and embarrassingly parallel MCMC parallelise one fit by partitioning a single dataset and merging subset posteriors as a product of densities; a 2024 IRT application does this by randomly partitioning examinees and merging with a Wasserstein barycenter while holding identification constraints fixed, so no linking is needed. Periodic recalibration of an evolving bank is a different problem: applied there, those methods would re-analyze all data at each update rather than combine a new period with already-computed earlier posteriors. The authors also choose aggregation over sequential updating — carrying each period forward as the next period's prior — for three stated reasons: fitting each period as a self-contained hierarchical model lets its own estimated population prior stabilize its sparse items rather than a prior fixed in advance; the model specification, including the parameter-prediction model, may legitimately change between periods; and aligning periods through a transformation fitted to anchors collectively requires only aggregate anchor invariance, which is weaker than the item-level invariance that reuse of prior item parameters presumes.

Method: Per-Draw Linking and Two-Layer Consensus

Linking. Each period's fit identifies the latent metric only up to a linear transformation, θ_ref = A_m θ + B_m with A_m > 0 and B_m period- and dimension-specific, so periods are not directly comparable. The constants are chosen to minimize a weighted discrepancy between reference and transformed expected-score (characteristic) curves over an anchor set — a Haebara criterion with a pseudo-Huber loss, a smooth surrogate for the least-absolute (L1) criterion of robust Haebara linking, evaluated by quadrature, with each anchor weighted in proportion to the number of responses on which it was calibrated so that better-estimated items contribute more to the link. Critically, the criterion is minimized separately for each posterior draw, following Baldwin's strategy, which turns the collection of per-draw constants into a sample from the posterior of the linking constants and thereby propagates linking uncertainty — together with its dependence on the item parameters — into the linked posteriors. Two anchor filters guard the link: items whose potential scale reduction factor exceeds 1.05 in either period are removed, and an iterative purification refits a provisional mean–sigma link and drops anchors in the top decile of absolute residuals for at most five iterations.

Aggregation. Linked period posteriors are summarized as Gaussians and combined as a precision-weighted product, with the per-period population prior subtracted out and a single replacement prior reinstated: Λ_c = Σ Λ_m − Σ Ω_m + Ω⋆ and b_c = Σ Λ_m µ̂_m − Σ Ω_m m_m + Ω⋆ m⋆. Each period's posterior incorporates its own estimated prior, so the naive product counts that prior M times and is over-concentrated; the correction targets posterior dispersion, not only location. When periods share a fixed prior, the expression reduces to the Bayesian committee machine rule Λ_c = Σ Λ_m − (M − 1)Ω — so the general form is required precisely because this prior is estimated and differs across periods. The replacement prior is itself built by consensus at the population layer: per-period posteriors of the feature-regression coefficients are combined as a Gaussian product with a weak zero-mean hyperprior counted once, and the consensus residual covariance is the item-count-weighted Fréchet mean of per-period covariances taken in log-Cholesky coordinates, which is positive definite by construction and avoids the determinant inflation of an entrywise average.

Eligibility. The subtraction can render Λ_c non-positive-definite for items the period data barely identify beyond the population model, or whose Gaussian summaries are noisy. Rather than regularising by pseudo-inversion — which would impute dispersion the data do not support — those items are flagged and assigned the population prediction ξ_j ~ N(X_j β⋆, Σ⋆). The proportion routed to this fallback is presented as a diagnostic of whether the periods are individually informative enough for the procedure at all.

Assumptions. Four are stated plainly: disjoint examinee samples, so period likelihoods are conditionally independent and the pooled likelihood factors; simple structure, with each item loading on a single latent dimension, which reduces metric indeterminacy to the linear map; aggregate anchor invariance; and Gaussian summaries of each period's item posterior, prior contribution and population-coefficient posterior.

Empirical Evaluation: Data and Protocol

The procedure was applied to operational response data from the Duolingo English Test, measuring two latent dimensions with items in several response formats (selected-response, partial-credit and continuous). Each period was calibrated by a scalable Bayesian explanatory IRT engine with period-specific item posteriors thinned to approximately 1,000 draws. The run comprised four consecutive quarterly calibration periods spanning about a year, with the third period defining the reference metric; each period included on the order of tens of thousands of items and hundreds of thousands of examinees. Linking used 200 draws of each non-reference period, and the consensus Gaussian was sampled to give 2,000 draws per item.

The benchmark is a pooled single-run fit, deliberately configured to avoid confounding: it uses a separate ability distribution per period — a multiple-group model in which each period's examinees form a group — matching the ability structure of the per-period calibrations. All comparisons are restricted to converged parameters, with an (item, parameter) pair entering only if its split-chain potential scale reduction factor stays at or below 1.1 in the pooled run and in every period run containing the item; that screen removed 1.9 percent of pairs. Three comparisons are reported: recovery of item-parameter posterior means and standard deviations, an uncertainty-by-exposure diagnostic, and agreement of the ability distributions.

Key Findings

1. Posterior means agreed closely between the consensus aggregate and the pooled benchmark: r = .998 with RMSE 0.129 for the difficulty parameter d, and r = .991 with RMSE 0.081 for log a₁ — evidence that per-draw linking and precision-weighted aggregation introduce no systematic distortion of location.
2. Posterior standard deviations also agreed: r = .970 with RMSE 0.066 for d and r = .920 with RMSE 0.036 for log a₁, showing that the prior correction recovers most of the pooled posterior's dispersion, not merely its center.
3. The uncertainty-by-exposure diagnostic behaved as the theory predicts but left a residual. Mean ratios of consensus to pooled posterior SD were 0.91 / 0.94 / 0.94 for d and 0.96 / 0.95 / 0.98 for log a₁ across low, mid and high per-period exposure tertiles — mild under-dispersion at every exposure level, worst in the lowest tertile, which is the signature of residue from prior over-counting rather than of added linking variability.
4. Linking was independently corroborated at the ability level: per-period ability means implied by the linked calibrations differed by at most 0.08 reference-metric units, and the ratio of corresponding ability standard deviations ranged from 0.98 to 1.07.
5. The update-cost claim is architectural rather than measured: because each period is calibrated once and combined with already-computed earlier posteriors, the cost of an update scales with the new period rather than the accumulated history, and the full history need not be held in memory at once.

What this means for practice

• Researchers. Aggregate separately calibrated periods with per-draw linking. Solving the characteristic-curve criterion for every posterior draw reproduced pooled item parameters at r = .998 (RMSE 0.129) for difficulty and r = .991 for log a₁ in this application, so linking uncertainty can be carried inside the item posteriors rather than fixed.
• Designers. Enforce disjoint examinee samples across calibration periods before aggregating. Conditional independence fails where retakes place the same candidates in two periods, and the empirical evaluation satisfied that condition rather than probing its violation.
• Designers. Propagate linking error into the parameters that drive adaptive delivery. Per-draw linking is what brought consensus posterior standard deviations into agreement with the pooled benchmark, r = .970 for difficulty and r = .920 for log a₁.
• Designers. Check posterior dispersion by exposure tertile and treat sparse items accordingly. Consensus standard deviations ran 0.91 to 0.94 of pooled values for difficulty across low, mid and high exposure levels, with the worst under-dispersion in the lowest tertile.
• Researchers. Report the fallback rate every cycle. Items whose consensus covariance fails to stay positive definite are routed to the population prediction, and the authors present that proportion as the diagnostic of whether periods are individually informative enough for the method at all; their own run does not report it.

Limitations

• Disjoint samples are a requirement, not a detail. Cross-period conditional independence assumes disjoint examinee samples, since substantial overlap counts some responses more than once - a real constraint in an assessment where candidates can retake, satisfied by the empirical evaluation but something a general deployment must actively enforce.
• An approximation whose adequacy is demonstrated, not proved. The combined factors are plug-in Gaussian summaries of each period's hierarchical posterior, assessed empirically through the benchmark comparison.
• Marginal, not joint, posteriors. The aggregation targets each item's marginal posterior, so the joint dependence induced by shared hyperparameters and common linking constants is not reconstructed: marginal posteriors are what adaptive item selection and scoring consume, but they are not a substitute for a full joint posterior where a downstream use depends on correlation structure across items.
• What the study does not establish. It demonstrates agreement with a pooled benchmark on one operational assessment, for two item parameters, where the anchor and convergence screens were met and the periods were individually informative enough that the fallback rate was tolerable; it does not report the proportion of items routed to the population fallback, does not compare against sequential updating, and does not test behavior under period-to-period specification changes. Nor does it address the exposure-control and fairness questions that automatic generation raises on the content side, where a bank that grows by prediction instead of pretesting changes what kind of evidence supports an item's difficulty.

Why It Matters for AI-Era Assessment

The paper's significance sits at the junction of psychometrically aware AI and operational scale. Feature-based parameter prediction is increasingly how AI-generated items enter scoring, and the authors' framing tightens what such a system must promise: calibration should refine those predictions rather than discard them, since the population model aggregated at its second layer is exactly the regression of item parameters on item features that prediction estimates. Their consensus population coefficients are therefore a re-estimate of the prediction model itself, which means each recalibration cycle improves the population layer that sparse new items borrow strength from.

The methodological point generalizes beyond language assessment. Any continuously evolving bank — an adaptive tutor's item pool, a competency-based program's assessment library, a university's growing question bank — faces the same three-way tension between calibration cost, memory, and statistical fidelity, and the standard advice (refit everything periodically) quietly becomes infeasible as generation outpaces pretesting. Divide-and-conquer Bayesian computation has been applied within a single calibration before; the contribution here is identifying precisely why the evolving-bank case breaks the existing machinery — unlinked metrics across periods, and an estimated, hierarchical prior whose multiplicity cannot be handled by the usual device of raising each subset's prior to the power 1/M — and supplying the two-layer correction that resolves both. Treating item parameters as estimates carrying uncertainty, and propagating linking error into that uncertainty rather than treating the transformation as fixed, is the same standard the validity and linking-error literatures have argued for in the frequentist case; consensus calibration is what that standard looks like when the calibration itself is Bayesian.

For assessment programs, the practical takeaway is that recalibration frequency need no longer be rationed by compute. If an update scales with new data, banks can be re-linked and re-aggregated on a quarterly or even faster cadence without a full-history refit — which matters most exactly where adaptive delivery and fast item turnover make stale parameters most costly.

Connected Concepts

• Item Response Theory — hierarchical Bayesian calibration of item parameters at bank scale
• Educational Measurement — linking, equating and posterior uncertainty as the core objects
• Psychometrically Aware AI — AI-generated items carried by calibrated, uncertainty-aware parameters
• Automated Question Generation — the generation pipeline whose output volume creates the calibration problem
• Assessment Validity — a bank whose parameters come from prediction rather than pretesting
• Large Language Models (LLMs) — language models behind automatic item generation and difficulty prediction
• Educational NLP — text-based and embedding-based difficulty estimation feeding the population model
• Language Learning — the Duolingo English Test as the operational setting
• Learning Analytics — scalable computation over large sparse adaptive response streams
• Adaptive Learning — adaptive selection and scoring depend on the item posteriors produced

Connected Articles

• Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach — Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach
• The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory — The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory
• Multimodal Item Parameter Estimation using Simulated Response Probabilities — Multimodal Item Parameter Estimation using Simulated Response Probabilities
• Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study — Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study
• From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations — From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations
• Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms — Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms
• Applying IRT to distinguish between human and generative AI responses to multiple-choice assessments — Applying IRT to distinguish between human and generative AI responses to multiple-choice assessments
• A Decade of Reflection and Thematic Review on Artificial Intelligence's Impact on Educational Measurement — A Decade of Reflection and Thematic Review on Artificial Intelligence's Impact on Educational Measurement
• Towards Scalable Measurement of Durable Skills — Towards Scalable Measurement of Durable Skills

Citation

Jewsbury, P. A., Nydick, S. W., Liao, M., & Chen, S. (2026). Bayesian Consensus Calibration of Continuously Evolving IRT Item Banks. Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con), NCME.