Multimodal AI Tutoring in STEM

When LLMs process STEM problems that require interpreting diagrams, graphs, or schematics alongside text, their accuracy degrades substantially. This effect is:

General-purpose LLMs achieve near-ceiling accuracy (~96%) on text-only physics problems, but accuracy drops to ~74% on image-rich multimodal problems due to the Multimodal Interference Effect. A simple structured dialogue intervention—requiring no retraining—recovers performance to 95%.^Syal Multimodal Dialogue STEM 2026

The Multimodal Interference Effect

When LLMs process STEM problems that require interpreting diagrams, graphs, or schematics alongside text, their accuracy degrades substantially. This effect is:

Consistent across models: Claude, Gemini, and ChatGPT all showed the drop

Dominantly visual: Visual Processing Errors were the most prevalent failure mode

An equity concern: Students working on image-rich problems—often those most needing support—receive less reliable AI assistance

Error Taxonomy

Syal et al. (2026) identified four empirically-derived failure modes on OpenStax physics problems:

Category	Prevalence	Definition
Visual Processing Errors	Highest	Failures to extract information from graphs, diagrams, or images
Context Misinterpretation	Moderate	Wrong physics concepts or problem framing assumptions
Mathematical Computation Errors	Low	Calculation or unit errors
Hybrid Errors	Moderate	Combinations of the above

Note: Visual and Hybrid errors appeared only in multimodal (not text-only) problems.^Syal Multimodal Dialogue STEM 2026

The Structured Dialogue Intervention

A three-step scaffolding protocol corrected 82% of all errors and 100% of Visual Processing Errors:

1. Elicit visual description: "Describe what you observe in the figure/graph"

2. Correct observable elements: Clarify visual properties (e.g., vector direction, graph intercept) without providing physics principles or formulas

3. Re-prompt for solution: "Solve the problem"

This relies purely on prompt design—no model retraining, no specialized benchmarks, no GPU access required. Under-resourced institutions can implement it immediately.

Relationship to Tutoring-Specific Design

The structured dialogue intervention is structurally similar to tutoring-specific guardrails: it forces the LLM to reason step-by-step rather than jumping to answers. It also preserves productive cognitive load by making the student (or in this case, the LLM) attend to visual structure before solving.^Syal Multimodal Dialogue STEM 2026

Implications

For STEM educators: Image-rich problems remain a high-risk use case for raw LLM tutoring; structured prompting protocols dramatically improve reliability

For tool designers: Visual reasoning pipelines should not be treated as a solved capability; interface-level scaffolding may matter as much as model scale

For equity: The Multimodal Interference Effect disproportionately harms students relying on free/public AI tools for STEM support

Open Questions

1. Does the intervention generalize to other STEM disciplines (chemistry, biology, engineering)?

2. What happens when the student directly implements the dialogue protocol vs. an intermediary system?

3. Do visual reasoning capabilities improve in newer model generations (e.g., GPT-5 multimodal)?

Connected Concepts

Teacher AI Competency

Socratic AI Dialogue

Help Seeking

Connected Articles

Syal Multimodal Dialogue STEM 2026

Tutoring Specific Vs General AI

Citation

tasks:, T.W.A.V.L.W.T.F.M. & Preview, P. (2026). Multimodal AI Tutoring in STEM