VIRA-Bench: Diagnosing Pedagogical Contingency Understanding in Online Tutoring Videos

Synthesis: Wang, Gu, Liu, Zhang, Gu, and Lu (2026) built a diagnostic Benchmark for a narrow capability they call visually grounded pedagogical contingency: reading a short tutoring exchange, not just naming its events. From a corpus of more than 1,000 recorded one-to-one online English tutoring sessions they assembled 1,005 clips of 15–60 seconds and 1,322 four-option questions across Observable Evidence Grounding, Interaction Contingency Reasoning, and Support-Response Modeling, anchored in the Initiation–Response–Follow-up structure of classroom discourse. Across 11 Multimodal AI models the best, Gemini 3 Pro, reaches 61.3% overall against a 91.0% human reference on a stratified 200-question subset, a 29.7-point gap. The released clips span 72 learners and 22 teachers, with every face blurred before distribution. Removing video lowers grounding most and disrupting event order lowers contingency reasoning most, which the authors present as distinct behavioral dependencies rather than one aggregate video-QA score.

Key Findings

1. A 29.7-point gap separates the best model from the human reference. Gemini 3 Pro reaches 61.3% overall (95% CI [58.7, 63.9]) against 91.0% for three experts majority-voting on a stratified 200-question subset.
2. The leading models cluster rather than rank cleanly. Five models fall between 55.9% and 61.3% with overlapping confidence intervals, while the remaining six range from 20.3% to 43.0%, so the authors report layer profiles rather than ranks.
3. Contingency reasoning is the weakest layer. Every top-tier model orders the layers SRM > OEG > ICR, and the human reference follows the same order (94.0% > 92.0% > 88.0%) at uniformly high accuracy.
4. Video removal hits grounding hardest. Dropping the video but keeping the transcript lowers OEG by 12.7–22.0 points across the five representative models, and ASR-only overall accuracy sits at 35.7–48.9%.
5. Order disruption hits contingency reasoning hardest. Permuting visual segments with ASR held fixed lowers ICR by 11.9–19.1 points, all surviving Holm correction, with Sequence Recognition falling 16–30 points.
6. Answer format does not explain the pattern. Free response lowers overall accuracy by 12.1 and 11.7 points, and a stem-plus-options condition collapses to 29.2–29.7% against 55.9–59.7% with full video.

The exchange, not the event

The construct is deliberately relational. A clip is scored as an exchange in which a teacher's visible move elicits a learner response and the teacher then adjusts; the question is whether a model can recover that chain rather than recognize each part. Two foundations carry the design: the Initiation–Response–Follow-up structure (Sinclair and Coulthard, 1975) and contingent teaching theory (Wood et al., 1976; van de Pol et al., 2010), the same evidence-driven adjustment logic that Scaffolding and Formative Assessment describe. Layers separate by cognitive operation, not difficulty. Observable Evidence Grounding (497 questions) asks for answers grounded in screen content, teacher action, or student behavior. Interaction Contingency Reasoning (529) asks whether a response followed, preceded, or merely coincided with an intervention. Support-Response Modeling (296) asks for a predicted response or a judgment of whether support matched the learner state. Eight subtasks split these operations, from Action Identification (215 questions) to Causal Attribution (82), and each 15–60 second clip (mean 32.4 s) is centered on a single intervention.

Items built to resist shortcuts

Construction keeps generation and evaluation apart. GPT-4o drafts a structured description of each clip — screen events, student behaviors, teacher speech acts, temporal dynamics, cognitive demand — and researchers audit every description against the video before any question exists, a step that dropped 7.4% of clips for unrecoverable evidence. GPT-5 then writes candidate items under constraints that hold option length within 20% of the mean, keep distractors concrete, and require that stem plus options be insufficient on their own. Educational-dialogue resources in Educational NLP typically rely on transcripts, which cannot show the screen operation at all. Two held-out models, GLM-5V-Turbo and Claude-Haiku-4.5, answer each item from stem and options alone; items both answer correctly count as surface shortcuts and are removed, cutting 10.6% of candidates, and a full-input pass removed 34.4% more. Three experts then reviewed a 360-item set (300 random, 60 high-risk), accepting 340 unchanged, revising 16, and removing 4 — 94.4% direct acceptance, Fleiss' κ = 0.76.

Eleven models against a stratified human ceiling

All 11 models answered zero-shot with video sampled at one frame per second plus a separate ASR field, with option order shuffled per item. Gemini 3 Pro led at 61.3% and Qwen3-VL-32B-Inst trailed at 20.3%, below the 25% random baseline; three models scored below chance, and the authors show this was not a parsing artifact, since predicted labels spanned A–D at 22.0–26.6% each with zero invalid outputs. The reference came from three experts with graduate training in applied linguistics who answered the same 200 questions (25 per subtask) after watching the clips: 175, 170, and 177 out of 200 individually, 91.0% by majority vote. Because every model answered all 1,322 questions while humans answered 200, the paper warns the subset result "should not be read as a precise human estimate for all 1,322 questions," though its 95% lower bound of 86.2% still sits about 25 points above the best model. For Video in Education, the consequence is blunt: these systems do not yet read the exchange the way a teacher does.

What the perturbations do and do not show

The controls carry the argument. With video removed, OEG fell 12.7–22.0 points and every OEG decline survived Holm correction, with Action Identification dropping 16–25 points and Content Recognition 23–40, while Behavior Extraction fell only 4–9 because talk carries part of that signal. Under visual shuffle with ASR fixed, ICR fell 11.9–19.1 points, and four of five OEG changes and four of five SRM changes were not significant. Shuffling video and ASR together, which preserves local cross-modal alignment, still cost ICR 21.5 points for Gemini 3 Flash and 19.7 for Qwen3.6-Plus while OEG moved 6.9 and 0.6, so video–text mismatch cannot explain the decline. Surface complexity does not either: ICR stems are the shortest at 13.0 words, option lengths match the other layers, and clip durations differ by at most 1.1 seconds. What the design cannot support is a claim about mechanism or tutoring skill: these are behavioral dependencies on input, and the protocol scores answers about recorded contingency, not the open-ended generation, action, or multi-turn adaptation tutoring demands — a distinction that matters for anyone applying these results in evaluation work.

What this means for practice

• Instructors. Do not read a benchmark score as readiness to delegate tutoring. The best of 11 models answered 61.3% overall against a 91.0% reference set by three experts on a 200-question subset, and on judging whether support matched the learner it sat 29.1 points below that reference.
• Instructors. When recorded tutoring video is used for professional learning, keep the judgment with the teacher; the study shows models can be queried about an exchange without showing they can recognize a confused learner in the moment.
• Educational technology developers. Publish layer-level scores and perturbation deltas alongside aggregate accuracy. The five leading models span only 55.9–61.3% with overlapping intervals, and one number hides whether grounding, order reasoning, or support judgment moved.
• Researchers. Reuse the paired-perturbation design — ASR-only, fixed-ASR visual shuffle, synchronized shuffle — before claiming a model reasons about interaction, and treat the 200-question subset as a reference point rather than a ceiling for all 1,322 items.

Limitations

• No learners or teachers took part as research participants. The study analyzes 1,005 clips from 72 learners and 22 teachers, recorded by a commercial institution between 2018 and 2021 under a data-sharing agreement, so no new instruction was observed and no learning outcome measured.
• Human verification rests on small expert panels: three auditors judged a 360-item set (300 random, 60 high-risk) at 94.4% direct acceptance with Fleiss' κ = 0.76, and three other experts answered the 200-question stratified subset (25 per subtask) at 91.0% by majority vote, 95% interval [86.2, 94.2].
• Coverage is 15–60 second excerpts of one-to-one online English tutoring with mostly Mandarin-L1 learners; the authors state findings may not transfer to multi-party, cross-domain, or longer-horizon interaction.
• The protocol scores four-option answers, not open-ended generation, action, or multi-turn adaptation, so a high score is not evidence that a model would tutor well.
• The paper reports no evaluation date or collection window, all 11 models are provider-default zero-shot calls at one frame per second, and evidence sufficiency had the weakest inter-annotator agreement of the four audit dimensions (κ = 0.68).

Citation

Wang, W., Gu, H., Liu, G., Zhang, P., Gu, N., & Lu, T. (2026). VIRA-Bench: Diagnosing Pedagogical Contingency Understanding in Online Tutoring Videos. arXiv preprint arXiv:2610.29428 (cs.HC).