Probing AI-Generated Physics Solutions and Preparing Students to Critique Them

Synthesis: Borse et al. (2026) examine AI-generated physics solutions from two connected angles: how prompt design shapes solution quality and how students can be prepared to critique those solutions. Using a rotational-mechanics problem and the Minnesota Assessment of Problem Solving (MAPS) rubric, they show well-specified prompts improve completeness while underspecified and multimodal prompts expose weaknesses in physics reasoning. In a student-evaluation phase with 24 introductory physics lab groups, MAPS-guided reflection produced more expert-aligned critiques than problem-solving-only training, which yielded uncritical or misconception-based assessments. The study grounds both model-reasoning benchmarks and student AI Literacy training in physics education research.

Prompt Design Shapes Solution Quality

Adapting a problem-classification framework, the authors evaluated OpenAI o4-mini responses with the MAPS rubric across prompt variations. Well-specified prompts improved solution completeness, whereas underspecified and multimodal prompts exposed weaknesses in physics reasoning and correctness — evidence that Prompt Engineering quality directly determines the reliability of AI-generated solutions students encounter.

Preparing Students to Critique

Twenty-four introductory physics lab groups evaluated an o4-mini solution either after independently solving a related problem or after critiquing the AI-generated solution with MAPS-based reflection questions. Problem-solving-only groups produced uncritical or misconception-based critiques; MAPS-guided groups identified expert-aligned issues including skipped numerical procedures and undefined notation. This shows critique training is a scaffoldable Critical Thinking skill rather than an automatic byproduct of problem-solving experience.

Implications for Physics AIED

The approach integrates model-reasoning benchmarks with pedagogical intervention, connecting Physics Education, Metacognition, and AI Literacy. Reflection rubrics turn AI fallibility into a learning resource, supporting Research Methods AIED work on evaluating both AI outputs and students' evaluation of them.

Connected Concepts

Connected Articles

Citation

Borse, N. S., Bralin, A., Savage, S., & Rebello, N. S. (2026). Probing AI-generated physics solutions and preparing students to critique them. arXiv:2608.12533.