Multimodal Item Parameter Estimation using Simulated Response Probabilities

Synthesis: This paper fine-tunes a multimodal large language model (Qwen3.5-based) to reconstruct multiple-choice model (MCM) and three-parameter logistic (3PL) item characteristic curves. By learning to reproduce students' systematic error patterns across a range of ability levels, the LLM implicitly captures underlying response probabilities and can approximate item difficulty on held-out test items directly from predicted option probabilities.

Key Findings

1. LLMs can learn IRT curves implicitly. A multimodal LLM prompted and fine-tuned to replicate choice probabilities across a corpus of image-and-text multiple-choice items, conditioned on labeled ability levels, learns the response patterns encoded in 3PL and MCM curves.

2. Direct difficulty estimation. Item difficulty on a held-out test set can be approximated directly from the model's predicted option probabilities, without explicit parameter-fitting.

3. Multimodal relevance. Because items contain both image and text stimuli, the approach extends IRT-style estimation to multimodal assessment items.

Implications

For Item Response Theory and Educational Measurement, this suggests a path to calibrate assessment items using LLMs that emulate student responding, which is relevant to Psychometrically Aware AI and to pre-testing items before deployment. It connects to Student Modeling and Automated Assessment work where models must reason about how learners of different abilities respond.

The approach has practical implications for Automated Question Generation pipelines and for scaling item calibration in LLM-based assessment systems, though the work is presented as a technical proof-of-concept and would benefit from broader validation across item types and populations.

Connected Concepts

Connected Articles

Citation

Ormerod, C., & Kim, Y. (2026). Multimodal item parameter estimation using simulated response probabilities. arXiv:2608.10154. cs.CL.