Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis

Synthesis: Strugatski, Zeinfeld, and Alexandron (2026) ask whether assessment instruments measure the same underlying constructs for humans and LLMs, arguing that validity-based inference from Large Language Models (LLMs) performance on human-designed exams requires similarity in the latent structure of responses. Using a case study across two STEM instruments — a high-school chemistry diagnostic and the quantitative-reasoning section of a university entrance exam — they compare human response data with responses from six Multimodal AI LLMs (OpenAI GPT-4o/GPT-5.2, Google Gemini 1.5 Pro/3 Pro, Anthropic Claude 3.5 Sonnet/4.5) using exploratory factor analysis, factor congruence, and resampling. They find systematic differences between human and LLM factor structures — LLM–human similarity reliably fell below the human–human baseline across both instruments. The findings call into question the validity of evaluation practices that use educational assessments to make generalizable claims about AI capabilities.

The validity condition for LLM evaluation

A common approach to evaluating LLMs is to administer assessment instruments designed to measure skills and competencies in humans — standardized exams, diagnostics, entrance tests — and treat performance as evidence about the LLM's underlying ability on the same constructs. From a validity perspective, however, such inferences require that the relationship between observed performance and the underlying construct, established for humans, also holds for LLMs. A necessary condition for transferring score interpretations is similarity in the latent structure of responses to the assessment. This study directly tests that condition.

Method: comparing latent structures

The authors adopt a case study design across two educational contexts: (1) a high-school chemistry diagnostic completed by several hundred Grade 11–12 students, and (2) a standalone quantitative-reasoning section from a high-stakes university entrance examination. LLM responses were generated from six multimodal models across three families. Because the assessments included figures, formulas, and visual content, multimodal input was required. The analysis combined:

• Exploratory factor analysis (EFA) to extract latent factor structures from each group's responses;
• Factor congruence to quantify similarity between human and LLM structures, using a matching algorithm (Hungarian method on absolute congruence) repeated over resamples;
• Resampling baselines comparing LLM–human (LH) congruence against the human–human (HH) congruence distribution as the empirical baseline of expected structural agreement.

Key findings

• Systematic structural differences: Across both instruments, humans and LLMs produced substantially different factor structures — LLM–human similarity remained considerably lower than human–human similarity, measured via mean matched factor congruence. This was consistent across instruments and choices of the number of factors.
• Factor retention diverges: Parallel analysis retained different numbers of factors for humans vs. LLMs in both datasets (e.g., in Chemistry humans consistently retained five factors while LLMs most often retained four; in Quantitative Reasoning humans retained 7–8 vs. LLMs' five). The Kaiser criterion happened to match within each instrument, but parallel analysis revealed the divergence.
• HH baseline is itself variable: Human–human similarity was far from perfect — distributions were broad rather than concentrated near 1.0 — so the main result is the consistent gap between HH and LH matching, not perfect human reproducibility.
• Implication: Established assessment instruments may capture substantially different constructs in humans and LLMs, so performance on them is weak evidence for claims about LLM abilities on the same skills the instruments were designed to measure in humans.

Significance for psychometrically aware AI

This paper contributes a validity-oriented framework for comparing latent structures across humans and LLMs, complementing the knowledge base's work on psychometrically aware AI. It shows that even when an LLM scores well on an exam, the score may not be interpretable the way a human score is — because the latent construct structure is different. This is a deeper form of Assessment Validity concern than raw score comparisons: it applies to Educational Measurement practices, AI evaluation using human benchmarks, and any AI-based assessment that assumes human validity evidence transfers. For LLM benchmarks built from human assessments, the implication is that benchmark scores may index a different underlying structure for the AI than for the humans the items were normed on.

What this means for practice

• Researchers. Test the latent structure before reusing an instrument on an LLM: fit EFA separately for humans and models and compare factor congruence against a human–human baseline, because in both instruments LLM–human similarity stayed below the human–human distribution and parallel analysis retained different factor counts (chemistry: five for humans versus four for LLMs; quantitative reasoning: 7–8 versus five).
• Researchers. Report the human–human agreement distribution alongside any human–LLM similarity figure: human–human congruence was broad rather than concentrated near 1.0, so a single value cannot be read as close without that reference.
• Assessment professionals. Stop reading high LLM scores on human-designed exams as evidence about the constructs those items were normed for; until structural similarity is established, the score indexes a different latent structure.
• Assessment professionals. Check factor retention with parallel analysis rather than the Kaiser criterion alone: the two agreed within each instrument here only coincidentally, and the human–LLM divergence appeared only under parallel analysis.

Limitations

• Case-study scope: two instruments (a high-school chemistry diagnostic taken by 931 students, and a quantitative-reasoning section from a high-stakes university entrance exam) and six multimodal LLMs (GPT-4o/GPT-5.2, Gemini 1.5 Pro/3 Pro, Claude 3.5 Sonnet/4.5); the authors state external validity is limited by this small number of instruments and tools.
• Comparisons used resampled sets of 120 human respondents and 120 LLM responses per dataset, and the LLM sample is relatively small compared with the human sample; models from different families were pooled as a single population despite undisclosed architectural differences.
• Non-public datasets and proprietary systems constrain replication and leave an unresolved contamination risk: the private assessment instrument had to be uploaded to proprietary tools, data are available only on request, and the authors cannot rule out that the instrument influenced later model versions.
• Only the internal-structure aspect of validity is tested; generalization to related tasks that measure the same underlying abilities is left untested, and prompting strategies, generation parameters, and interfaces can bias LLM response patterns even when standardized.

Citation

Strugatski, A., Zeinfeld, L., & Alexandron, G. (2026). Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis. [cs.HC].