LumiNote: LLM-Assisted Multimodal Instruction for VR Stage Lighting Education

Synthesis: Stage lighting instruction requires instructors to hold four layers together at once — aesthetic intent, fixture-level operations, visible stage effects, and learner-facing explanations — and existing VR systems simulate lighting hardware without supporting that live teaching work. Liang et al. build LumiNote, an Large Language Models (LLMs)-assisted VR system that turns spoken pedagogical intent, anchored by a laser pointer, into instructor-reviewable spatial annotations, executable lighting demonstrations, and real-time jargon explanations. Across 55 instructor prompts and 531 generated actions, assistance proved most valuable for expressive, under-specified goals (visual effects: 212 of 245 actions applied) and weakest for fixture-specific requests, where 26 of 28 rejections traced to directional-reference misreads such as "left light." Instructors used suggestions as a controllable refinement process — 86.4% of rejected or modified actions triggered a new prompt rather than manual adjustment — shifting effort from manual demonstration setup toward pedagogical expression while keeping authority over the lesson. The study's central caution is a representation mismatch: the cues instructors rated most useful for externalising expert reasoning were not the cues novices found easiest to follow.

The Instructional Problem: Bridging Intent, Operation, and Representation

Technical theater arts combine artistic expression with complex technical operations, demanding instruction that tightly integrates conceptual explanation with hands-on demonstration. In stage lighting education, instructors routinely move among aesthetic intent, fixture-level decisions, visible stage effects, and learner-facing explanations — and within VR teaching environments they face "instructional friction" coordinating spatial manipulations, technical configurations, and in-situ explanations. Existing research and commercial previsualization tools (Vectorworks Spotlight, Capture, WYSIWYG) reproduce fixtures, effects, and operational workflows, and Simulation removes dependence on scarce venues and equipment, but they support lighting design rather than the instructional work of connecting technical operations with spatial behavior, perceptual effects, and design intent. The authors name a recurring translation challenge: instructors must translate pedagogical intent into concrete scene manipulations while simultaneously making those manipulations understandable to learners. Prior immersive systems support structured practice, procedural learning, and authored tutorials; comparatively little work addresses instructors' spontaneous work during live teaching, leaving experts to bridge both gaps unaided. This framing places the problem squarely in instructional design, not in hardware fidelity.

Design Requirements from a Formative Study with Four Experts

Semi-structured 60-minute interviews with four experts (two stage lighting instructors with 5 and 3.5 years of experience, two theater arts professors with 15 and 7 years) were coded inductively to a Cohen's Kappa of 0.8927. Participants described reliance on 2D previsualization, LMS-based feedback, and face-to-face discussion, and noted that 2D tools cannot reproduce the spatial scale and embodied viewpoint needed to judge lighting in a real venue. They also stressed that realism alone is insufficient: if VR only replicates physical scenarios, operational complexity and physical discomfort can outweigh immersion, so VR must offer functional advantages beyond Simulation — making invisible lighting relationships visible, accelerating live demonstrations, and supporting feedback that is hard to deliver physically. Three requirements follow: DR1, situated spatial annotation that makes spatial relationships and operational cues visible in context; DR2, on-demand real-time lighting demonstrations that preserve instructional flow without manual setup delays; and DR3, linguistic support that translates domain-specific jargon into learner-accessible explanations. To ground DR1 and DR2, the authors analyzed five beginner lighting courses and distilled three recurring strategies — spatial geometry (arrows and lines for direction and coverage), contextual tagging (labels linking jargon to physical components), and actionable demos (real-time parameter changes) — which become the representational vocabulary of the system.

LumiNote: From Spoken Intent to Reviewable Scene Actions

LumiNote is an instructor-facing system built inside a 1:1 digital twin of a theater hall with a functional lighting grid, implemented in Unity (6000.0.23f1) on Meta Quest 3, using Whisper for speech-to-text and GPT-4o for intent interpretation, backed by Django and SQLite. The workflow is deliberately narrow: the instructor taps "Record" in the LLM Interaction Panel, speaks a teaching instruction (for example, that contrasting colors can create a harmonic effect), and the system transcribes the speech, interprets instructional intent, and returns a list of suggested actions — changing spotlight colors, adjusting intensity, rotating a light toward a performer, showing a beam-direction arrow, or adding a concept tag. Before any change is applied, the instructor can apply, reject, or manually refine each suggestion, or apply all / decline all in bulk; action cards are populated only after generation, so nothing touches the stage unapproved. The system generates three families of aid: spatial geometry (3D direction lines and beam vectors tracing a spotlight's path to its target), contextual tagging (semantic tags at world positions, such as "Upstage" or "Cyclorama", plus jumping-star highlights), and actionable demos on spotlights (power and dimmer percentages, RGB color tuning, beam angle and inner/outer ring softness, and spatial targeting). Every applied adjustment also creates real-time console tagging — labels pinned to the exact virtual buttons and sliders with the operation name and target value — creating an explicit link between the visible effect and the technical input on a professional lighting desk. A separate jargon pipeline transcribes instruction, highlights domain terms such as "hue" and "color wheel", and shows context-aware definitions beside the transcript, so the lesson need not stop for basic definitions. Instructor mode additionally exposes session recording, global reset, and a broadcastable 3D laser ray described as a "virtual baton"; student mode allows manual operation and jargon review but cannot trigger LLM actions, so learners only see expert-validated output. This is Multimodal AI interaction in its literal sense — voice, controller pointing, and scene state fused into one request — and it functions as Scaffolding for the instructor rather than for the learner.

Constrained Grounding as the Safety Mechanism

Rather than letting the model drive the virtual theater, LumiNote bounds generation along three dimensions: what can be changed, what can be referenced, and how changes are represented. A scene serialization module writes a lightweight JSON snapshot of fixture IDs, emission states, transforms, color, intensity, beam angles, and barn-door openness, plus performer and instrument positions and zoned stage areas; an iterative laser anchor resolves deictic references such as "rotate the spotlight to here" by asking the user to point with the controller and appending the resulting intersection coordinate to the metadata. The prompt then combines runtime scene context, static scene metadata describing layout and fixture groups, and few-shot instructional examples paired with a predefined action schema; the authors use direct few-shot in-context learning rather than multi-turn reasoning to keep latency low in VR, and the examples cover four foundational categories — fixture/configuration, spatial behavior, visual effects, and composition strategies — mapping pedagogical intent onto the executable action vocabulary without additional model training. Ambiguity is handled at the schema level: an instruction like "make the performer stand out with a warmer light" becomes an executable color adjustment such as SET_COLOR(Light_27,[255,140,0]). Crucially, the constraints "do not guarantee that the generated actions are technically or pedagogically appropriate" — they only establish a bounded proposal space, from which a validation module filters unsupported actions, references, and parameter values before anything reaches the instructor. The remaining suggestions pass through a Human-in-the-Loop interface for review, modification, application, or rejection, which the authors frame as Guardrails rather than intelligence: the point is inspectability, not autonomy.

Exploratory Study: Three Instructors, Twenty-Four Learners

The evaluation is two-phase. In Phase 1, three instructors (2M, 1F) taught the same four beginner topics — side lighting, color dimensions, color combinations, and emotional atmosphere — first in a No LLM VR session and then with full LumiNote support, after a 15–20 minute onboarding; the design deliberately places the manual session first so that LLM assistance is observed against each instructor's own flow, and the authors acknowledge this fixed order as a limitation. Participants spanned beginner, intermediate, and advanced VR familiarity and differed in teaching experience (one year, four years, three years). In Phase 2, 24 students (10M, 14F), aged 18–31 (M = 22.62, SD = 4.15) and overwhelmingly unfamiliar with the concepts, were randomly assigned to a No LLM or With LLM recording created by the same instructor, then completed three focused tasks and one open-ended composition task for a solo singer, submitting a scene screenshot and a verbal rationale scored by expert instructors on concept accuracy, visual atmosphere, and comprehensiveness. Instructor evidence came primarily from interviews, session recordings, and observed behavior, with SUS, NASA-TLX, custom ratings, and session duration used descriptively given n = 3; learner evidence came from interviews and open-ended responses alongside feature ratings, SUS, NASA-TLX, IPQ presence, learning self-ratings, completion time, and expert scores. All interaction-log measures are explicitly framed as interaction-level indicators of whether a suggestion entered the teaching flow, not as measures of model accuracy — a methodological caution worth carrying into other Large Language Models (LLMs)-in-classroom studies.

Where Assistance Helped Most — and Where Grounding Broke Down

Across the three With LLM sessions, instructors issued 55 unique prompts producing 531 prompt–suggested action pairs: 380 (71.6%) applied, 90 (16.9%) rejected, 61 (11.5%) modified. Visual effects formed the largest prompt category (27 of 55, 49.1%) with high adoption (212 of 245 actions, 86.5%), and composition strategies reached 77.5% pooled adoption; the two together accounted for 38 of 55 prompts. Fixture/configuration showed the lowest pooled direct adoption (38 of 66, 57.6%) — but this was strongly instructor-dependent, with T1 applying 27 of 28 such actions while T3 applied 10 of 37. The instructive detail sits inside the failures: of 28 rejected fixture/configuration actions, 26 involved directional-reference interpretation, where the model selected the wrong fixture for phrases like "left light" or "side light". Interviews match the logs — T2 valued LumiNote "when I wanted to create a lighting effect that was not very explicit and was more feeling-based", estimating roughly 70% alignment, while all three instructors reported speed gains for presenting an intended visual effect quickly. The authors' reading is that generative value lies at an intent-to-operation gap: experts know what they want to communicate but have not articulated every action needed to realize it, so assistance matters most when intent is expressive and under-specified, and precision requests instead demand reference disambiguation, spatial anchoring, and parameter confirmation. Their design implication is adaptive support keyed to intent specificity, treating generation and grounding as complementary rather than interchangeable.

Controllable Refinement, Not Autonomous Instruction

Instructors did not treat generated suggestions as finished teaching plans. T3 insisted the lesson still follow a planned flow with an overall goal divided into subgoals, skipping outputs that did not match the intended meaning or reprompting for a closer configuration. Follow-up behavior quantifies this: among 147 rejected or modified suggestions with recorded follow-up, 127 (86.4%) were followed by a new prompt, 19 (12.9%) by no action, and only 1 (0.7%) by direct manual adjustment — reprompting, not hands-on fixing, was the dominant corrective move, commonly triggered by insufficiently specific instructions, spatial-reference problems, or speech-transcription issues. All three instructors asked for templates or preset lighting combinations tied to concepts, a pull toward structured, modular input rather than open conversation, consistent with broader work on Prompt Engineering and teacher control over AI generation. Use patterns also split by experience: T1, with one year of teaching, triggered far more generated actions (26 prompts, 397 actions) than T2 and T3 (21 prompts / 67 actions and 8 prompts / 67 actions), with T2 suggesting the tool may help instructors still building familiarity with their materials — an observation the authors flag as suggestive rather than a systematic expertise effect. The effort shift is corroborated descriptively: NASA-TLX fell from 3.50 (SD = 1.88) to 2.22 (SD = 1.11), mean session duration from 19 min 34 s to 12 min 33 s, and SUS rose from 65.83 (SD = 16.27) to 79.17 (SD = 7.64), with ratings up for immediacy, clarity, instructional effectiveness, professionalism, and pedagogical capability. The authors are careful that this redistribution did not remove decision making — it moved instructor effort away from low-level configuration toward selecting, evaluating, and refining candidate demonstrations.

The Representation-Alignment Gap Between Experts and Novices

The sharpest finding is a mismatch between expert-facing and learner-facing representations. Instructors rated AI-generated lighting demonstrations and directional arrows as the most effective functions, both at 6.0 on a seven-point scale, and described arrows as externalising relationships such as light direction and beam behavior that are hard to communicate verbally; they rated console action tags and the animated instructor avatar lower, both 5.0. Students inverted the ranking: the green laser pointer was most helpful (M = 5.50, SD = 1.17), followed by console panel tagging with suggested actions and values (M = 5.33, SD = 1.15) and lighting demonstrations (M = 5.25, SD = 0.87), while spatial arrow annotations and the avatar were moderate (both M = 4.58) and star-shaped highlights ranked lowest (M = 4.33, SD = 1.30). Learner comments point to why: one student noted the theater "is large, it's very hard to intuitively understand which part to focus on, so laser can help", and another that tagging on buttons and sliders showed "what to do exactly". Novices needed representations that answer where to look, what changed, and which operation produced the change — attention guidance and operation linkage, not a reproduction of the expert's spatial reasoning. Outcome measures reinforce the caution: presence was higher in the With LLM condition (M = 5.72, SD = 0.82) than without (M = 4.93, SD = 0.80; t(22) = −2.40, p = .025, d = 0.98), and 8 of 12 With-LLM students described specific techniques such as "blurring the inner ring to reduce hardness" versus 9 of 12 No-LLM students leaning on abstract descriptions like "personal feeling", yet SUS, NASA-TLX, learning self-ratings, completion time, and expert task scores showed no significant differences. The authors therefore position the LLM as a potential mediation layer that preserves the instructor's intended concept while translating its visual and technical presentation for learners, and they note that longer-term study is needed to know whether richer representations affect retention or Transfer of Learning rather than only articulation. The four transfer conditions they distil — under-specified expert intent, executable environmental state, a bounded operational vocabulary, and retained expert authority — generalize the pattern to other immersive domains such as chemistry demonstration, with the explicit caveat that technical executability in a Simulation does not imply pedagogical appropriateness.

Connected Concepts

• Virtual and Augmented Reality — the instructor-facing immersive teaching environment
• Multimodal AI — fused voice, pointing, and scene-state input
• Human-in-the-Loop — generated actions remain proposals subject to expert approval
• Generative AI — operational realization of under-specified instructional intent
• Large Language Models (LLMs) — intent interpretation, action schema mapping, and jargon explanation
• Human AI Collaboration — iterative prompt-refine interaction with a co-participant
• instructional design — design requirements derived from instructor workflows
• Teaching — instructor authority over what enters the lesson
• Intelligent Tutoring — LLM assistance embedded in live pedagogical delivery
• Simulation — the digital twin of a theater hall and its lighting grid
• Visualization — spatial arrows, tagging, and beam vectors as instructional representations
• Experiential Learning — hands-on lighting practice in a risk-free venue
• Embodied Learning — spatial scale, viewpoint, and physical pointing in the venue
• Situated Learning — explanation anchored in the current scene state
• Scaffolding — jargon explanations and demonstrations that hold instructional flow
• Arts, Design and Media Education

Connected Articles

• Generative AI and Extended Reality in Collaborative Architectural Design Education: An Exploratory Studio Study — Generative AI and Extended Reality in Collaborative Architectural Design Education: An Exploratory Studio Study
• Design and Implementation of a Real-time Multi-site Immersive Learning System Using Photon Fusion — Design and Implementation of a Real-time Multi-site Immersive Learning System Using Photon Fusion
• Development and applications of Generative AI in architectural design studios — Development and applications of Generative AI in architectural design studios
• Cultivating Design Creativity of Vocational Students: A Model of Project-Based Learning in AI-Enabled Immersive Virtual Environments — Cultivating Design Creativity of Vocational Students: A Model of Project-Based Learning in AI-Enabled Immersive Virtual Environments
• When Should Teachers Control AI Generation for Mathematics Visuals? — When Should Teachers Control AI Generation for Mathematics Visuals?
• From Prompt to Embodied Simulation: Using Generative AI to Create AR Physics Learning Tools — From Prompt to Embodied Simulation: Using Generative AI to Create AR Physics Learning Tools
• Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy — Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
• Multimodal Dialogue in STEM Education — Multimodal AI Tutoring in STEM
• Code as Anchor, Memory and Metaphor as Support: Learner Experiences with Multi-View Visualizations — Code as Anchor, Memory and Metaphor as Support: Learner Experiences with Multi-View Visualizations

Citation

Liang, D., Li, C. Y., Wei, Z., Xu, X., Xia, M., Qu, H., & Tong, W. (2026). LumiNote: LLM-Assisted Multimodal Instruction for VR Stage Lighting Education. arXiv preprint.