OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora

Synthesis: Chen et al. (2026) introduce OmniPhys, a large-scale Multimodal AI benchmark for physics understanding and reasoning built from Chinese educational corpora, spanning middle-school through university-level problems. The benchmark comprises 15,246 questions and 19,850 images with fine-grained annotations supporting analysis of reasoning processes and knowledge usage, and — unusually — systematically evaluates multimodal output generation, including models' ability to synthesize structured physics diagrams. Extensive evaluations reveal critical gaps in current multimodal LLMs, especially in complex reasoning and visual generation, positioning OmniPhys as a foundational resource for advancing multimodal intelligence in physics and for informing automated physics Assessment.

Why physics needs a unified multimodal benchmark

Multimodal large language models (MLLMs) have demonstrated strong abilities across visual and textual reasoning tasks, but their development in the physics domain is hindered by the lack of a comprehensive benchmark. Physics is a prototypical multimodal arena, demanding rigorous integration of textual descriptions, visual diagrams, and symbolic logic for accurate reasoning. Existing physics datasets rarely satisfy three critical criteria simultaneously: (1) cross-stage knowledge fusion spanning middle school to university; (2) multimodal input comprehension requiring interpretation of complex textual and visual cues; and (3) multimodal output generation assessing the model's ability to actively synthesize diagrams rather than merely select options. OmniPhys addresses this gap by unifying all three in a single benchmark.

Benchmark design

OmniPhys is a Chinese benchmark designed to assess physics mastery from secondary education to university levels, covering five major physical disciplines including mechanics, electromagnetism, and optics. All questions are curated from contemporary examination papers and authoritative textbooks, undergoing a strict multi-stage filtering process to guarantee difficulty and pedagogical validity. The benchmark includes:

• 15,246 questions and 19,850 images, with detailed annotations supporting fine-grained analysis of reasoning processes and knowledge usage.
• A multimodal output generation subset that assesses MLLMs' capabilities in physics diagram understanding and editing — a fundamental component of authentic physics problem solving that most benchmarks omit.
• Coverage spanning question types from multiple-choice to open-ended problem solving, grounded in authentic Assessment material.

Evaluation findings

Comprehensive baseline evaluations reveal that despite recent advances, current multimodal LLMs exhibit significant capability gaps in the physics domain, especially in complex reasoning and visual generation. The multimodal output tasks — where models must synthesize or edit structured physics diagrams — proved particularly challenging, underscoring that generating authentic physics representations remains an open problem for MLLMs. These findings have direct implications for whether generative AI systems can serve as reliable partners in physics learning and automated assessment, connecting to broader knowledge base evidence that AI systems still struggle with the specialized, multimodal, and diagram-heavy tasks characteristic of authentic STEM assessment.

Connection to the broader knowledge base

OmniPhys matters to the knowledge base for three reasons. First, it extends the physics-education evidence base on AI capability with a large, authentic, Chinese-educational-corpus benchmark — complementing studies of AI performance on physics problems and LLM support for computational thinking in physics. Second, its emphasis on diagram generation connects to multimodal learning and authentic physics problem solving, where the ability to construct representations is as important as selecting answers. Third, its finding that MLLMs struggle on complex reasoning and visual generation informs realistic expectations for tutoring and automated assessment in physics, supporting the knowledge base's recurring theme that AI excels at routine tasks but underperforms on the authentic, high-level reasoning that defines deep disciplinary learning.

What this means for practice

• Designers. Evaluate physics models on multimodal output, not answer selection alone: the diagram-editing subset showed that even frontier image-generation models systematically violate physical laws, and leading models remained below 70% strict mastery.
• Designers. Report a text-only and a caption-substituted baseline beside the multimodal condition, because on the adversarial Test-Mini subset some architectures scored higher without the diagram, suggesting visual inputs were read as distractor noise.
• Researchers. Treat monolingual benchmark scores as confounded when instructions are in English and the problems are in Chinese, since weaker performance may reflect Chinese multimodal alignment rather than physics reasoning.
• Researchers. Budget human evaluation for generated physics diagrams: MLLM-based judges overestimated output quality and overlooked subtle physical inconsistencies despite the structured rubric.

Limitations

• The corpus is predominantly Chinese educational material, and the cross-lingual prompting protocol (English instructions, Chinese problems) leaves physics reasoning ability confounded with Chinese multimodal alignment; the authors plan A-Level and IPhO material to separate the two.
• Multimodal output evaluation remains underexplored and no fully scalable solution exists: MLLM-based judges overestimated the quality of generated content and missed subtle physical inconsistencies, necessitating costly human evaluation.
• The ablation study ran on a Test-Mini subset of 10% of the data, selected by consensus of five state-of-the-art models for a 75% empirical failure rate, not on the full benchmark.
• Failure analysis is limited to qualitative observations of representative error patterns; the authors state they still lack an error-attribution study at scale that would separate visual perception failures from logical reasoning errors.

Citation

Chen, H., Lin, Y., Yushanjiang, N., Lin, X., & Zhang, M. (2026). OmniPhys: A unified multimodal benchmark for physics understanding and generation from Chinese educational corpora. arXiv:2608.25398.