Multimodal Learning with Generative AI

📄 Full text: Liverpool Repository · local

A comprehensive educator's guide to integrating Generative AI into multimodal teaching, learning, and assessment across higher education. Built on Kress's social semiotic theory of multimodality, the guide positions GenAI as a 'cyber-social' partner that complements—but cannot replace—human meaning-making. It proposes the MMLD-AI unifying model (UDL + ABC Learning Design) and the Dual-Track Cyber-Social Learning Model for designing effective human-AI collaboration.^varga-atkins-educators-guide-multimodal-learning-genai-2025

Core Position: Pragmatic, Not Uncritical

The guide adopts a middle way between "techno-fixing" and rejecting AI as an existential threat. It argues that:

GenAI is already embedded in daily life; ignoring it does students a disservice

GenAI is not intelligent (no consciousness, understanding, or ethical judgement)

Human educators and learners bring vision, purpose, nuanced critique, and meaning-making that AI cannot replicate

Effective use requires cyber-social partnership: humans and machines with complementary strengths

The Four Costs of GenAI

Cost Domain	Key Concern	Educational Response
Individual	Privacy, data protection, equity of access, mental health, over-reliance	Transparent authorship; approved tools list; scaffolded critical engagement
Environment	Image/video/audio generation uses significantly more energy than text	Mindful use; limit iterations; group demonstrations; digital decluttering
Knowledge	Removes sourcing process integral to retention; short-term gains may displace deep learning	Students clarify own understanding before consulting AI; metacognitive scaffolding
Future Jobs	Entry-level white-collar roles vulnerable to automation	Focus on human strengths: contextual reasoning, ethical judgement, craftsmanship

AI Literacy in Multimodal Contexts

Three Levels

1. Basic literacy — Awareness of multimodal GenAI platforms, capabilities, and appropriate uses (creating prompts, generating visual outputs) 2. Intermediate literacy — Co-create multimodal content, critically evaluate AI outputs, scaffold uses (transform lecture notes into visuals or podcasts) 3. Advanced literacy — Design activities/assessments incorporating multimodal GenAI; lead ethical and philosophical discussions

Four Implementation Scales

Scale	Strategies
Individual	Workshops on creative multimodal tasks; prompt crafting practice; reflective assignments documenting AI use
Module	Embed GenAI literacy into learning outcomes; optional multimodal tasks with clear rubrics; creative/reflective critique components
Programme	Cross-module policies; consistency and transparency via workshops and discussion; alignment with graduate attributes (criticality, creativity, digital fluency)
Institutional	Clear policies with checklists; vetted tools; data privacy protocols enforced; avoid rigid mandates in favor of flexible guidance

The MMLD-AI Unifying Model

The Multimodal Learning Design with GenAI model merges:

Universal Design for Learning (UDL): multiple means of engagement, representation, and action/expression

ABC Learning Design: storyboarding the student journey through learning types

Six Multimodal Engagement Types (adapted from ABC)

1. Acquisition of information 2. Investigation and/or research 3. Collaboration with others 4. Production of artefacts (learning, teaching, or assessment) 5. Practice of approaches/theories/principles/skills 6. Discussion/discourse, including critique/evaluation

For each engagement type, educators decide on multimodal affordances of GenAI and explore respective cyber-social strengths.

The Dual-Track Cyber-Social Learning Model (Galla et al., 2025)

This complementary model maps human vs. AI strengths across Bloom's taxonomy processes:

Process	Human Strengths	AI Strengths	Cyber-Social Approach
Knowledge & framing	Contextual understanding; embodied knowledge; critical verification	Rapid data retrieval; pattern recognition; broad topical coverage	Humans define purpose and frame problems; AI generates background data and inspiration; humans filter and verify
Interpretation & analysis	Causal reasoning; cultural/ethical awareness; implicit meaning	Correlation analysis; theme identification; feature extraction at scale	AI identifies statistical patterns; humans determine causality, relevance, deeper significance
Application & prototyping	Situated judgement; adaptive problem-solving; ethical decision-making; craftsmanship	Rapid simulation; consistent rule application; code/digital artefact generation	AI generates digital prototypes; humans adapt for real-world complexity, apply physical craft, ensure ethics
Synthesis & creation	Novel conceptual blending; purpose-driven integration	Cross-domain pattern integration; combinatorial exploration	AI explores possible combinations; humans evaluate, refine, and integrate meaningfully

Practical Integration: Three Strands

Teaching (Educator-Created Content)

Generating visuals, diagrams, and infographics from text prompts

Creating podcast scripts and video summaries

Building interactive simulations and virtual scenarios

Using AI to get feedback on marking rubrics and assessment briefs

Learning (Student-Created Content)

Students transform lecture notes into multimodal artefacts (visuals, podcasts, videos)

Collaborative group projects using GenAI for brainstorming and prototyping

Critical evaluation: students annotate AI-generated outputs for accuracy, bias, coherence

Ethical protocols establishing clear boundaries (e.g., "do not use AI to write reflections; do use it for brainstorming visuals")

Assessment and Feedback

Multimodal assessment: students submit artefacts combining text, image, audio, video

AI-assisted peer and self-assessment with structured rubrics

Educators use GenAI to generate formative feedback at scale, then verify and personalize

Transparent: assessment briefs explicitly state when and how GenAI may be used

Relationship to Existing Research

Guide Principle	Wiki Connection
Cyber-social partnership (complementary strengths)	principled-ai-education — "AI must augment, not displace" aligns perfectly
Four costs framework (individual, environment, knowledge, jobs)	ai-tutor-safety-harms — Costs to knowledge overlap with cognitive offloading; environmental costs are a new dimension
AI literacy levels and scales	ai-literacy — ICAP framework; collaborative learning; this guide adds institutional scaling and multimodal specificity
MMLD-AI model (UDL + ABC + six engagement types)	adaptive-learning-systems — Multi-resolution personalization; agentic-workflows-education — Planning and reflection paradigms
Dual-Track Cyber-Social Model	pedagogical-llm-training — Reward "guiding" over "answering"; human-in-the-loop-ai — Human verification of AI outputs
Multimodal assessment redesign	authentic-assessment — Six-dimensional framework; formative-assessment — AI-generated feedback with human validation
Scaffolding and metacognition	self-regulated-learning — UDL's emphasis on student agency; metacognition — Cyber-social metacognitive awareness
Faculty development across four scales	faculty-development-genai — CTL pragmatic transition model; this guide adds module-level and programme-level strategies

Case Study Themes from the Guide

The guide includes 15+ educator case studies spanning:

Healthcare: AI avatars for patient communication training (H5P interactive scenarios)

Bioscience: Multimodal groupwork designing organisms for future Earth scenarios

Business/HR: Peer conflict resolution with AI-generated scenarios and video avatars

Education/Teacher training: AI visual metaphors for reflective practice

Chemistry: AI-generated molecular visualizations and 3D models

Languages: Text-to-speech and avatar creation for pronunciation practice

General: Explainer videos, digital posters, podcast scripts, interactive quizzes

Open Questions

1. Environmental cost awareness: How can educators and students make informed trade-offs between the pedagogical value of multimodal GenAI artefacts and their energy costs? 2. Transfer across modalities: Does competence in AI-assisted multimodal creation in one domain (e.g., visual design) transfer to another (e.g., audio production)? 3. Assessment validity: When students use GenAI to create multimodal assessment artefacts, how can assessors distinguish genuine human meaning-making from AI-generated polish? 4. Scaling the MMLD-AI model: Can the six engagement types be operationalized as automatic learning design recommendations, or does human pedagogical judgment remain essential?

dot-framework-survey

multimodal-ai-feedback-learning — Zhao et al.: concrete implementation of multimodal AI feedback system validating the MMLD-AI framework

ai-literacy — Collaborative learning and ICAP framework; this guide adds multimodal and institutional dimensions

faculty-development-genai — CTL-level professional learning; this guide adds module/programme/institutional scaling

principled-ai-education — Goals-models-technologies framework; this guide provides concrete multimodal implementation

authentic-assessment — Assessment redesign for AI-present contexts; this guide adds multimodal artefact assessment

formative-assessment — AI feedback generation; this guide adds multimodal feedback strategies

self-regulated-learning — UDL and student agency; this guide adds cyber-social regulation

metacognition — Metacognitive awareness of human vs. AI strengths/weaknesses

pedagogical-llm-training — Training for guiding behavior; this guide adds multimodal output training

human-in-the-loop-ai — Educator verification of AI outputs; this guide adds peer and self-assessment loops

adaptive-learning-systems — Personalized content; this guide adds multimodal personalisation

agentic-workflows-education — Planning and reflection paradigms as engagement types

ai-tutor-safety-harms — Costs to knowledge overlap; environmental costs expand the taxonomy

educational-llm-alignment — Alignment between AI capabilities and pedagogical goals

multimodal-ai-tutoring — Multimodal errors and corrections; this guide focuses on productive multimodal use

collaborative-ai-tutoring — Group-level cyber-social collaboration

socratic-ai-dialogue — Discussion/discourse as one of six engagement types

Sources

Varga-Atkins, T., Saunders, S., Beckingham, S., Hartley, P., Keshishi, N., Lacković, N., Li, N., Lindsay, R., Wen, R., & Winder, I. (2025). An Educators' Guide to Multimodal Learning and Generative AI. University of Liverpool. PDF