Multimodal AI

Multimodal AI — AI systems that process, understand, or generate content across multiple modalities — text, images, audio, video, and structured data — and the educational questions these systems raise. In AI in education, multimodal AI appears in three distinct roles: as the learning content learners create and engage with (multimodal learning), as the capability boundary of tutoring systems that must interpret diagrams and graphs (multimodal tutoring), and as the assessment signal used to evaluate understanding (multimodal measurement).

Questions to Consider

• Think of a graph, force diagram, or schematic you've ever struggled to explain in words. What does that experience suggest about the limits of a text-only AI tutor trying to help with image-rich problems?
• A physics tutor answers text-based problems ~96% of the time but drops to ~74% on problems that embed meaning in diagrams. Before reading, what do you think causes this 'multimodal interference'—and can you think of a fix that doesn't involve retraining the model?
• You've likely generated both text and images with AI tools. Have you found that 'prompting for pictures' differs from prompting for text? What skills might students need to translate an abstract idea into a precise visual prompt?
• Multimodal AI can grade essays, generate feedback with audio narration, and even reconstruct exam item statistics from image-and-text items. What does the shift from text-only to multimodal assessment signal (or risk) for fairness and validity?
• How could the fact that AI support is less reliable on the very diagram-heavy problems that build deep STEM understanding create an equity gap between learners? Who is most affected?
• Multimodal systems can translate text to audio or visuals to support inclusive learning, but they also enable fine-grained classroom sensing. Where is the line between helpful multimodal access and surveillance?

Introduction

Multimodality in AI refers to the capacity to work across different representational forms rather than text alone. Modern Generative AI and Large Language Models (LLMs) systems increasingly accept and produce images, audio, and video in addition to text, opening new possibilities and new risks for education. Grounded in social semiotic theory, which holds that meaning is made across modes — not just words — multimodal AI changes how teaching, learning, and assessment are designed and evaluated.(Multimodal Learning with Generative AI)

Three faces of multimodal AI in education

1. Multimodal learning and content creation

Multimodal AI enables learners to produce and engage with content across text, image, audio, and video. An educator's guide to multimodal learning with generative AI positions these tools as a "cyber-social" partner: they complement — but cannot replace — human meaning-making.(Multimodal Learning with Generative AI)

• AI literacy in multimodal contexts is layered: basic awareness of multimodal platforms, intermediate co-creation and critical evaluation of outputs, and advanced design of multimodal activities and assessments.(Multimodal Learning with Generative AI)
• Multimodal prompting is itself a demanding epistemic practice. Students who prompt for images as well as text discover that "prompt literacy is different between prompting for text than it is for pictures" — translating abstract meaning into machine-readable multimodal prompts requires a precise visual vocabulary and exposes system limitations and bias.(Students' multimodal prompting practices as epistemic work in AI literacy development)
• Multimodal assessment shifts from essays to artifacts combining text, image, audio, and video, with educators using AI to scaffold creation and feedback rather than replace the learner's own production.(Multimodal Learning with Generative AI)
• Modality choice is an educational judgment, not a container. Elkington (2026) argues that choosing among written, oral, digital and multimodal formats decides what becomes possible for learners and visible to assessors, and counts controlled choice among modalities as a generative constraint rather than an administrative detail.
• Learner multimodal composing as a critical-thinking scaffold carries a trade-off. Lu et al. (2027) show that having upper-primary students turn written narratives into AI-generated images and short videos supported sustained gains in interpretation, analysis, evaluation, and explanation — but not inference. Because the visuals made story meaning explicit, students reported less need to infer implicit meaning from text alone; peer collaboration, not the multimodal tool, restored occasions for inference. Multimodal AI's value as a meaning-making partner is thus dimension-specific and depends on instructional design that deliberately re-introduces the inferential and self-regulatory work the externalization can short-circuit.
• Learner multimodal composition as critical AI literacy. Burriss et al. (2026) analyze 22 eleventh graders' 90-second to 3-minute video public service announcements on self-chosen AI Ethics issues — surveillance through school-regulated laptops and electronic "hall passes," informed consent, and punitive algorithmic accusation — as critical AI literacy enacted through composing across moving image, sound, text, and students' own bodies. Across all seven films harm was portrayed as emerging from human–machine entanglement rather than from the tool alone (an anthropomorphized "AI stalker" was played by a human actor in three of seven), and 15 of 18 end-of-unit responses said composing changed their understanding of AI ethics. The authors argue multimodal products both demonstrate and communicate critical competence — productive artifacts, reflections, and civic discourse can serve as Assessment evidence that text-only literacy scales structurally miss.

2. Multimodal tutoring and the capability boundary

When LLM-based tutors must solve problems that embed meaning in graphs, force diagrams, schematics, or tables, their accuracy degrades sharply — the Multimodal Interference Effect.(Multimodal Dialogue in STEM Education)

• On OpenStax physics problems, text-only accuracy of ~96% drops to ~74% on image-rich problems, consistently across model families.(Multimodal Dialogue in STEM Education)

• Visual Processing Errors — failures to extract information from graphs or diagrams — dominate the error taxonomy and are the most correctable failure mode.

• A simple structured-dialogue intervention (have the model describe what it sees, correct only observable misreadings without giving away physics, then re-prompt) restores accuracy to ~95% with zero retraining.(Multimodal Dialogue in STEM Education)

• This is an equity concern: students working on image-rich problems — precisely the problems that build deep conceptual understanding in STEM — currently receive less reliable AI support than those on text-only exercises.

• Constructing a visual aid is harder than reading one. On GeoVAD-Bench, supplying an expert auxiliary diagram raised accuracy (+3.3 to +7.0 points), but letting models construct their own auxiliary line widened the gap by 10.0 to 13.5 points — two models scored worse than with no visual reasoning at all (Dong et al., 2026).

• The boundary is a profile, not a level — and artistic imagery sits outside the region models handle well. MUSE (Zhu et al., 2026) evaluates 30 open- and proprietary VLMs on 12 tasks over 1,174 commissioned artworks, and the capability spread across dimensions is wider than any aggregate score suggests: scene classification is near-mature (23 of 30 models above 75.0, median 81.0). Meanwhile emotion detection tops out at 39.5, and the open-ended tasks that require models to articulate their evidence score at 50.90 (visual clue identification) and 49.18 (emotion cause inference) on semantic similarity. Compositional and viewpoint-dependent reasoning fail hardest — where the ground truth specifies no definite lateral or vertical relation, 90.0% and 73.3% of models assert one anyway, only 43.3% place the girl correctly in depth, and no model resolves all three dimensions of a single item. Failures also cascade: a mis-grounded character is then justified with a fluent rationale built from nearby visual semantics (butterflies, birds), which is the outcome most dangerous in tutoring because the explanation reads as competent. For image-based language learning this argues for dimension-level validation on the imagery a course actually uses, rather than importing a general multimodal score, and for extending the grounding checkpoint described below — describe what is seen, and where, before reasoning from it — to situated artistic content (MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education).

• The ceiling is low on curriculum-authentic figure-dependent math - and refusal is the calibrated fallback. Croteau & Heffernan (2025) evaluated six multimodal models on 376 image-Required middle-school items whose figures carry task-critical constraints. With images, the reasoning models o4-mini solved 57.7% of items and o3 52.4%, versus 34.0%-42.6% for the non-reasoning family; when the figure was withheld the models refused rather than guessed in 95.4%-98.9% of runs - an abstention pattern a tutor can treat as a missing-context signal rather than a learner error.

The practical design implication is a visual grounding checkpoint in multimodal tutoring: a deliberate step where the system describes what it sees before attempting a solution, giving the student or a human supervisor a chance to correct perceptual errors.(Multimodal Dialogue in STEM Education)

Abreu et al. (2026) add a prior constraint: an equation, graph or unit can appear in a report yet never be retrieved from the processed document, so a disagreement with the instructor may be an extraction failure rather than a reasoning one — which makes submission format part of the assessment design.

3. Multimodal assessment and measurement

Multimodal AI broadens both the content of assessment and the signal used to score it.

• Multimodal feedback systems integrate structured text, slide references, and streaming audio narration. In one study, AI multimodal feedback matched educator feedback for learning while significantly outperforming it on student perceptions.(LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback)

• Multimodal item response estimation uses fine-tuned multimodal LLMs to reconstruct item characteristic curves (IRT / 3PL) directly from predicted option probabilities on image-and-text items, connecting multimodal AI to Educational Measurement and Item Response Theory.(Multimodal item parameter estimation using simulated response probabilities)

• Educational vision-language model evaluation and multimodal LLM literacy extend the field's evaluation toolkit to multimodal reasoning and Visualization.(The aftermath of DrawEduMath: Vision language models underperform with struggling students and misdiagnose errors)(Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy)

• Multimodal grading of handwritten chemistry exposes a format-dependent capability boundary: Cvengros & Kortemeyer graded a 296-student handwritten general-chemistry final page-by-page against rubric images with a multimodal, reasoning LLM, scoring textual answers and chemical-reaction equations reliably (normed F1 highest) but drawing and graphing worse than random — background grids visually distract AI vision and scientific diagrams/chemical structures remain hard to interpret — reinforcing that multimodal AI's vision is not robust to representation-heavy work and is best deployed with human deferral of graphical items (Assisting the grading of a handwritten general chemistry exam with artificial intelligence).

• Construct recognition and comparative judgment are separable skills. Ma et al. (2026) express six multimedia-learning principles as reversible slide edits plus visual-equivalence sham controls, and find both models recovered every operation, principle and repair (8/8) while severity calibration failed entirely (0/8) — a composite score would conceal which layer fails.

• Fairness gains can be validated into existence. A multimodal attention estimator beat a visual-only baseline only modestly, and its gender-targeted MAE-gap regularizer cut the validation gap from 0.02 to 0.005 yet increased the gap and worst-group error on held-out subjects — so subgroup-aware, repeated subject-level validation is required before deployment (Fragkiadakis et al. (2026)).

• Multimodal grading can reproduce a selection outcome even where item-level scoring lags. Grading 10,364 handwritten Olympiad and university pages, an LLM matched examiner totals at r = 0.93–0.96 and placed the same five students on the Olympiad team, yet part agreement reached 70%: second-reader evidence, not a scorer of record (Pathak et al. (2026)).

• Diagram generation is a capability frontier, not a solved one. On a 15,246-question physics benchmark that scores multimodal output, synthesizing or editing structured physics diagrams proved harder than answering, and leading models stayed below 70% strict mastery — evidence that visual production lags visual comprehension (Chen et al., 2026).

Multimodality has a second sense: fusing signals from the learner's body rather than from content, where eye tracking, EEG, electrodermal activity, heart-rate variability and camera pose feed estimates of attention, engagement and cognitive load. That sensing strand lives at Sensor and Gaze Analytics, and its validity and consent problems are not the ones multimodal content raises.

Multimodal AI for language and accessible learning

Multimodal systems also expand access and personalization. AI-guided audio-video learning tools adapt playback speed, produce multimodal video summaries, and support pronunciation practice.(AI-guided learning: Research on knowledge and skill acquisition support methods using deep learning audio-video processing techniques) Multimodal knowledge graphs reason across images and text for educational tasks,(Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning) and multimodal representations improve Inclusive Learning by translating information across modes (e.g., text to audio or visual). Domain applications include handwritten-math grading and diagnosis,(Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work) affective tutoring with multimodal signals,(An Interpretable Closed-Loop Intelligent Tutoring System for Multimodal Affective Feedback in Asynchronous Presentation Training)(MathBuddy: Affective Math Tutoring) text-to-image learning in specialized fields,(NuclearDiffusion: Text-to-Image Foundation Models for Learning Nuclear Energy Concepts) and privacy-aware multimodal classroom sensing.(Robust and Efficient Motion Reasoning for Privacy-Aware Classroom Incident Recognition) Bird (2026) demonstrates a text-internal form of multimodality: fusing a fine-tuned ELECTRA transformer with computational-linguistics feature analysis to classify English literature by UK Key Stage, where the fused model (F1 0.996) far surpassed every unimodal baseline — evidence that combining representational forms, even within text, can outperform single-model approaches.

Challenges and design implications

1. Close the multimodal gap. Multimodal tutoring systems should include visual grounding and structured-dialogue scaffolds rather than assuming vision capabilities are robust.(Multimodal Dialogue in STEM Education)
2. Treat multimodal prompting as a teachable skill. AI literacy curricula must address modality-specific prompting, coherence across modes, and critical evaluation of multimodal outputs.(Students' multimodal prompting practices as epistemic work in AI literacy development)
3. Preserve human meaning-making. Multimodal AI should augment, not replace, the learner's own construction and evaluation of meaning across modes.(Multimodal Learning with Generative AI)
4. Extend evaluation to multimodal validity. Assessment validity, bias, and reliability must be examined when AI scores or generates multimodal artifacts.(Multimodal item parameter estimation using simulated response probabilities)(AI Ed Evaluation)
5. Watch equity and privacy. Unreliable support on image-rich problems and the data demands of multimodal sensing both carry equity and privacy implications.(Multimodal Dialogue in STEM Education)(Robust and Efficient Motion Reasoning for Privacy-Aware Classroom Incident Recognition)
6. Match the pipeline to the content. A cybersecurity lab assistant's multimodal LLM handled dense visual slides better, while an OCR-plus-LLM pipeline delivered comparable instructional value on text-centric slides at significantly lower computational cost.(Patel et al. (2025))
7. Collecting several streams is not integrating them. A review of 23 AI-for-Embodied Learning studies sorted multimodal work into three levels — streams gathered in parallel, streams aligned on a shared timeline, streams fused inside one model that estimates the learner's state — and placed most studies at the first, pairing image with text or motion with text without ever joining them. One study aligned facial affect, gaze and face detection after the fact; the closest approach to fusion modeled joint states from wristband and skeletal signals with a Markov process, and ran after the session rather than informing the feedback learners received. No reviewed system fused its inputs during use (Kim and Han (2026)).

Common Misconceptions

"A multimodal model handles multimodal problems." Accuracy degrades sharply when meaning is carried by the figure rather than the text. A physics tutor answers text-based problems about 96% of the time but drops to about 74% on problems that embed meaning in diagrams — the Multimodal Interference Effect. Twenty-two points of capability disappear exactly where the disciplinary content lives.

"Adding an image to the prompt resolves the diagram problem." The input modality is not the bottleneck; the interference is. A model that receives the diagram and still reasons from the surrounding prose has been handed the information and has not used it, which is why the failure is invisible unless someone compares the two problem formats.

"Multimodal capability is a single ladder, so the best model overall is best for the image-rich task." Capability is task-specific, and a model that leads on general benchmarks can trail on force diagrams. For image-rich disciplines the relevant test is the one with the meaning embedded in the figure.

"Text-only tutoring limits are a nuisance rather than a curriculum problem." In disciplines where the diagram is the problem — physics, chemistry, geometry, engineering, medicine — a text-only tutor's competence boundary falls across the assessed content rather than at its edges.

Connected Concepts

• Generative AI
• Large Language Models (LLMs)
• Knowledge Graph
• Intelligent Tutoring
• AI Literacy
• Prompt Engineering
• Feedback
• Assessment
• Educational Measurement
• Item Response Theory
• Learner Modeling and Adaptive Instruction
• Socratic Method
• Scaffolding
• AI Ed Evaluation
• Benchmark
• Higher Education
• Equity
• Privacy
• STEM Education
• Inclusive Learning
• Technologies — Umbrella: AI technologies and techniques (models, LLM training, robotics, RAG, agentic)
• Virtual and Augmented Reality — gesture, voice and spatial input as learning channels
• Speech and Voice Technologies
• Arts, Design and Media Education
• Sensor and Gaze Analytics

Connected Articles

• Assessing what matters most – the shifting value proposition of human-centred work in assessment practice — Human-centred assessment, assemblages and generative constraints (Elkington 2026)
• "Young Scholar[s] on the Beat": Multimodal composition as a form of critical AI literacy pedagogy — Video PSA composition on AI ethics as critical AI literacy pedagogy (Burriss et al. 2026)
• Fairness-Aware Multimodal Transformer Modeling for Real-Time Student Attention Estimation — Fairness-Aware Multimodal Transformer Modeling for Real-Time Student Attention Estimation
• OmniPhys: A unified multimodal benchmark for physics understanding and generation from Chinese educational corpora
• The aftermath of DrawEduMath: Vision language models underperform with struggling students and misdiagnose errors — VLM performance on handwritten student math work (DrawEduMath, Lucy et al. 2026)
• Multimodal Learning with Generative AI — Educator's guide to multimodal learning with generative AI (MMLD-AI model)
• Multimodal Dialogue in STEM Education — The Multimodal Interference Effect and structured-dialogue recovery in STEM
• LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback — Multimodal AI feedback matches educators on learning, exceeds on perceptions
• Students' multimodal prompting practices as epistemic work in AI literacy development — Students' multimodal prompting as epistemic work in AI literacy
• Multimodal item parameter estimation using simulated response probabilities — Estimating IRT item parameters with multimodal LLMs
• AI-guided learning: Research on knowledge and skill acquisition support methods using deep learning audio-video processing techniques — AI-guided audio-video learning support
• Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning — Multimodal knowledge graphs for educational reasoning
• Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy — Multimodal LLM literacy for scientific visualization
• An Interpretable Closed-Loop Intelligent Tutoring System for Multimodal Affective Feedback in Asynchronous Presentation Training — Multimodal signals in affective intelligent tutoring
• MathBuddy: Affective Math Tutoring — Affective multimodal math tutoring
• Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work — LLM cognitive diagnosis of handwritten math
• NuclearDiffusion: Text-to-Image Foundation Models for Learning Nuclear Energy Concepts — Text-to-image learning in nuclear engineering education
• Robust and Efficient Motion Reasoning for Privacy-Aware Classroom Incident Recognition — Privacy-aware multimodal classroom sensing
• Integrating generative AI into cybersecurity education: A study of OCR and multimodal LLM-assisted instruction — Multimodal OCR instruction in cybersecurity education
• Benchmarking Multimodal Large Language Models for Educational Slide Auditing — CFES-P24: Benchmarking Multimodal LLMs for Slide Auditing
• Large-scale AI grading of handwritten physics assessments: Score agreement and Olympiad team selection outcomes — AI grading of handwritten physics assessments (Olympiad)
• More externalization, but less inference? Exploring changes in young learners' critical thinking during conversational AI bot-supported multimodal writing practice — Multimodal AI composing and critical thinking in primary writing (Lu et al. 2027)
• Assisting the grading of a handwritten general chemistry exam with artificial intelligence
• Beyond Generation and Accuracy: Diagnosing and Enhancing Visual Chain-of-Thought for Geometry Problem Solving — Beyond Generation and Accuracy: Diagnosing and Enhancing Visual Chain-of-Thought for Geometry Problem Solving
• MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education — MUSE: 12 tasks over 1,174 artworks show VLM capability as a dimension-specific profile, weakest in affective interpretation and viewpoint-dependent spatial reasoning (Zhu et al. 2026)
• AI-Assisted Assessment of Experimental Physics Laboratory Reports: Potential, Limitations, and Support for Teaching Practice — AI-Assisted Assessment of Experimental Physics Laboratory Reports: Potential, Limitations, and Support for Teaching Practice
• Artificial intelligence in embodied learning: A systematic literature review — AI in embodied learning mostly observes: sensing roles 62.5% of coded role tags, and no reviewed system fused multimodal streams while learners used it (Kim & Han 2026)