Virtual and Augmented Reality

Virtual and Augmented Reality (VR/AR) — the display and interaction layer through which learning environments are experienced: fully synthetic spaces in VR, and digital content overlaid on the physical world in AR and mixed reality. AI enters this layer in two directions. It authors it, since generative AI now turns a natural-language description into a working browser-based AR or VR learning tool that no longer requires a specialist developer. And it inhabits it, as agents, pedagogical agents and retrieved knowledge guide a learner hands-free inside the immersive environment. What the modality adds over a screen is presence and embodiment; what it costs is fidelity, hardware, and a body's tolerance for being there.

Questions to Consider

• Where does the line fall between what is being modeled and the surface it is shown on? Compare a desktop patient simulator with a VR field trip to a site students cannot visit. Which differences would you expect to change learning, and which might only be novelty?
• A pilot found students reported feeling wavelength and amplitude through a hand gesture more strongly than through a slider — but it measured perception, not achievement, with 29 students and no comparison group. How much should reported "feeling" count toward adopting a tool?
• If generative AI lets an instructor with no programming background build a working AR simulation in an afternoon, what new responsibilities follow for validating the physics, judging fidelity, and deciding whether it belongs in a course?
• In a review of AI-powered nursing simulation, AI matched human actors for structured communication but not for tactile and emotionally complex scenarios. How would you sequence practice so learners rehearse some parts with AI and others with people?
• One classroom VR study reported minimal motion sickness while the meta-analytic estimate for intelligent VR with students with disabilities was not statistically significant. What would you want measured before a program invests in headsets?

Introduction

Virtual and augmented reality is a modality, not a model: it is the layer through which a learning environment reaches the learner. That makes it a different axis from Simulation, which is what gets modeled in the first place. The two are often conflated, because immersive environments are common delivery vehicles for simulations, but they vary independently — a desktop patient simulator is simulation without VR, and an AR overlay on a real instrument is VR without simulation. Keeping the axes apart matters for design: deciding to make practice risk-free is a different decision from deciding to make it embodied, and the two carry different costs, different failure modes, and different evidence. VR/AR is the delivery layer for a simulation and a close relative of game-based learning; it operationalizes embodied and situated accounts of learning, and it sits inside experiential and Active Learning practice.

AI now shapes this modality from both ends. It authors immersive content, because generative AI converts a structured natural-language prompt into a runnable AR or VR artifact, collapsing the specialist programming skill that used to gate production. And it inhabits the environment, as agents, pedagogical agents and retrieved domain knowledge supply guidance inside it — hands-free, in real time, grounded in sources the learner cannot consult while wearing a headset.

What the modality adds

The case for VR/AR rests on presence and embodiment rather than on information delivery. A shared virtual space can also decouple co-presence from geography: a VR classroom built on a real-time synchronisation layer put teachers and students in the same room and manipulating the same 3D science materials regardless of site, targeting up to twenty participants. In a one-hour session with 10 students on a Meta Quest 3, it reported good usability on the System Usability Scale and minimal Simulation sickness, while its own weakness was UI consistency — and its designers deliberately avoided the earlier peer-to-peer architectures whose frame rate degraded as participants joined, a reminder that presence is bounded by engineering. More controlled comparisons are sobering about what the surface alone buys: a 24-participant study of engineering mechanics found that mixed-reality apps and physical toolkits raised engagement over classroom instruction, yet complex visualizations remained difficult for learners in every condition. Engagement is the reliable effect; understanding is not.

Generative AI as the authoring layer

The barrier that used to define who could build an immersive tool has largely fallen. Using a four-element prompt structure — tools, display, hand controls, optimization — a teacher or student with no coding background can generate a browser-based, hand-controlled AR physics simulation that runs as a single HTML file with nothing beyond a camera, then refine it by describing what went wrong in plain language. The gesture is familiar from touchscreens: pinch and spread tunes a physical quantity rather than zooming a picture, so opening the fingers vertically raises amplitude, and horizontally lengthens wavelength and shifts the lamp toward the red end of the spectrum. The same structure generalizes: the Coulomb field around a fingertip spread through the room, two hands as opposing charges, and the right-hand rule for magnetic force drawn on the learner's own hand — rendered deliberately unmirrored, since mirroring would invert the very rule being taught.

The pilot is encouraging and preliminary in equal measure. With 29 second-year medical-imaging students in an introductory radiation physics course, all 29 agreed the gesture helped them "feel" what wavelength is (mean 4.52), amplitude control scored highest (4.59), 93% found controlling the wave in the air natural, and 86% reported feeling more engaged and focused than in regular learning. The evidence, however, is perception-only, single-class, and without a comparison group — so the honest reading is that Prompt Engineering has removed the production barrier, which makes the underlying question about embodiment testable, not that it has been answered.

AI inside the immersive environment

The second direction is where AI stops authoring and starts teaching inside the headset, and it is where retrieval grounding becomes load-bearing. An agentic immersive training platform for high dose rate brachytherapy built a digital twin of the treatment suite — anatomically precise patient models, catheters, afterloaders, applicators — so trainees could see an applicator's spatial orientation relative to organs at risk, without a shielding room, a live radioactive source, or the privacy hazards of a physical pelvic exam. A knowledge-aware assistant grounded in clinical guidelines supplied hands-free guidance through a three-tier voice interface (headset microphone to backend transcription and intent analysis to spatialised speech), which removes controller dependency during intricate maneuvers. Technically it worked: 3–5 second end-to-end latency across 50 Monte Carlo runs, context recall above 0.93 and answer relevance 0.87 on 52 expert-authored question-answer pairs, with a medical embedding model improving answer completeness.

Its gaps define the current frontier of the pattern. Evaluation was objective metrics plus a single domain-expert user, not learners; there was no automated Assessment or adaptive feedback; and the assistant cannot exceed the documents it was given, so institution-specific protocols and rare scenarios sit outside its competence. In other words, the tutoring intelligence inside immersive environments is still mostly guidance, not measurement — and the same platform architecture, with model inference offloaded to a local GPU backend, is what makes hands-free guidance fast enough to be usable at all.

The clearest attempt to date at that missing measurement comes from a vocational design studio rather than a clinical setting. In a 12-week interior-design course, a VR environment (headsets plus a 3D modeling tool) with an LLM-backed assistant rendered as a digital human was compared against conventional project-based instruction, with the assistant assigned a distinct role per phase — resource recommendation and task decomposition, layered questioning with knowledge maps, simulated design effects and flaw detection, then discourse logging for the teacher (Jin et al., 2026). Across 63 valid responses the immersive-plus-agent condition produced significantly higher design ability (η²p = .138) and creative ability (η²p = .111), cognitive (d = 0.90) and behavioral (d = 0.75) engagement, and motivation (d = 0.74) and satisfaction (d = 0.69) — with reported cognitive load lower, not higher (d = −0.52), which the authors credit to the assistant cutting search and cross-disciplinary integration effort. Two limits keep this from settling the question: ideational novelty and affective engagement did not move, and every outcome is self-report, with no artifact ratings or headset logs.

The contrast with a comparable design-education deployment is instructive. An architectural studio using a GenAI plus multi-user XR pipeline found declining design Self-Efficacy confidence for the teams that used it and no advantage in blinded panel ratings. The difference between the two is less the hardware than the orchestration: the vocational study fixed the phase structure, the assistant's role in each phase, and the evaluation rubric before the intervention, and measured productive ability separately from ideational novelty.

The instructor-facing version of the same pattern is younger and more fragile. Liang et al. (2026) let a stage-lighting instructor speak their intent into a VR scene with a laser pointer as the spatial anchor, and had an Large Language Models (LLMs) turn it into instructor-reviewable spatial annotations, executable lighting demonstrations and on-demand jargon explanations. Over 55 prompts and 531 generated actions, assistance was strongest on expressive, under-specified goals (212 of 245 visual-effect actions applied, 86.5%) and weakest on fixture-level requests, where 26 of the 28 rejected actions traced to directional references such as "left light" that the model resolved to the wrong fixture — grounding in the scene's spatial reference frame, not executability, was the binding constraint. Instructors used suggestions as a controllable refinement process rather than an answer: 127 of the 147 rejected or modified actions (86.4%) were followed by a new prompt, and only one by manual adjustment. The study's most cautionary result is representational — annotations that externalised expert reasoning did not reliably align with what novices understood, so immersive AI instruction carries both the grounding problem of the tutoring cases above and a second one, that instructor competence and learner comprehension are not the same target.

What the evidence shows

The strongest evidence for VR/AR is comparative and discipline-specific. A meta-analysis of 33 experimental and quasi-experimental studies (N = 3,181) of emerging technologies in teaching English as a foreign language found a small-to-moderate overall effect (g = 0.38) and, within it, the largest effects for VR/AR, with gains rising by educational level and favoring productive skills (speaking, writing) over receptive ones.

The counter-evidence is just as informative. In the first meta-analysis of AI-based interventions for students with disabilities — 29 studies, 239 effect sizes, medium overall effect g = 0.588 — intelligent VR systems produced g = 0.528, not statistically significant, while computer software reached 0.959 and robots 0.509. Publication bias was present and trim-and-fill reduced the overall estimate to g = 0.269. The pattern is not that immersion fails, but that its effects are small, heterogeneous, and sensitive to how the particular intervention was designed and compared.

The clearest negative result in the knowledge base comes from a studio deployment rather than a comparison of modalities: in a 27-student architectural design studio, teams using a GenAI plus multi-user XR pipeline declined more in design Self-Efficacy confidence (β = −1.675) and outcome expectancy (β = −2.088) than teams working the normal course workflow, with no significant difference in expert panel ratings of their presentations (Xiao et al., 2026). The authors explain it as phase-dependent complementarity with real friction — GenAI for externalising tentative ideas, XR for spatial and scale evaluation — alongside control, dimensional-fidelity, shared-attention and motion-comfort problems, a reminder that immersive tooling adds interaction costs as well as capability.

Fidelity, presence and the authenticity gap

Where immersive practice is used for interpersonal and procedural skill, the limiting factor is not visual fidelity but felt authenticity. A mixed-methods review of AI-powered nursing simulation (19 studies, N = 1,253) found AI effective for cognitive knowledge and affective outcomes but inconsistent for complex psychomotor skills, and named the reason: an authenticity gap covering emotional resonance, nonverbal cue recognition, and tactile and physical-examination dimensions. Its practical recommendation is a stepped simulation continuum — AI is well suited to highly structured objectives such as foundational communication and history taking, while advanced psychomotor and emotionally complex scenarios belong with human-standardized patients and clinical placement. Technical instability compounds the problem: speech-recognition delays inject extraneous cognitive load and anxiety, which makes stability and latency design levers rather than implementation details.

The same logic explains why presence is not automatically good. Motion sickness, UI inconsistency, hardware cost, and uneven device access decide who can use an immersive environment at all, which is why the modality's equity questions connect to Accessibility and Inclusive Learning rather than sitting apart from them.

Connected Concepts

• Simulation — what immersive environments usually display; the model, not the modality
• Embodied Learning — the mechanism the modality is supposed to exploit
• Multimodal AI — gesture, voice and spatial input as learning channels
• Situated Learning
• Experiential Learning
• Game-Based Learning
• Workplace Learning — the setting where immersive practice is most established
• Generative AI — the new authoring layer
• Prompt Engineering — how non-programmers build and refine immersive tools
• Agentic AI
• Pedagogical Agent
• RAG (Retrieval-Augmented Generation) — grounding guidance inside the headset
• Intelligent Tutoring
• Visualization
• Medical and Health Professions Education
• Accessibility
• Inclusive Learning
• Trust Calibration — fidelity, and the learner's awareness of its limits
• Cognitive Offloading — latency and instability as extraneous load
• Edtech Platform — synchronisation, latency and multi-site presence
• Arts, Design and Media Education

Connected Articles

• From Prompt to Embodied Simulation: Using Generative AI to Create AR Physics Learning Tools — four-element prompt generating hand-controlled AR physics simulations; 29-student pilot, perception-only evidence
• Agentic AI-driven Immersive Simulation: A Knowledge-Aware Virtual Training Platform for High Dose Rate (HDR) Brachytherapy — agentic AI VR training platform with a RAG-grounded hands-free assistant; digital twin, latency and retrieval metrics
• Design and Implementation of a Real-time Multi-site Immersive Learning System Using Photon Fusion — real-time multi-site VR classroom; usability and VR-sickness outcomes, UI consistency as the weak point
• AI-Powered Simulation for Nursing Education: Mixed Methods Systematic Review — the authenticity gap and the stepped simulation continuum
• Visualizing Engineering Fundamentals: Design of Mixed Reality and Physical Toolkits for Effective Learning — mixed-reality apps vs physical toolkits vs classroom in engineering mechanics; engagement up, complex visualization still hard
• A Systematic Review of Emerging Technology Applications for Teaching English as a Foreign Language Across Different Educational Levels — TEFL meta-analysis where VR/AR produced the largest subgroup effects
• Let's CHAT About Artificial Intelligence for Students With Disabilities: A Systematic Literature Review — intelligent VR for students with disabilities: positive but not statistically significant
• Connecting Education with Reality: AI as a Catalyst for Situated Learning — lack of immersive tooling as a barrier to situated learning
• MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education — gamified medical training with AI
• Technology-Enhanced Tabletop Exercises for Cybersecurity Education: Lessons Learned — augmented tabletop scenarios in cybersecurity education
• Generative AI and Extended Reality in Collaborative Architectural Design Education: An Exploratory Studio Study — Generative AI and Extended Reality in Collaborative Architectural Design Education: An Exploratory Studio Study
• Cultivating Design Creativity of Vocational Students: A Model of Project-Based Learning in AI-Enabled Immersive Virtual Environments — AI-IVE-PBL: immersive VR design studio with an LLM-backed teaching assistant, evaluated against traditional PBL (Jin et al. 2026)
• LumiNote: LLM-Assisted Multimodal Instruction for VR Stage Lighting Education — LumiNote: LLM-assisted multimodal instruction in VR stage lighting education, and where grounding broke down (Liang et al. 2026)