Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents

Synthesis: Lin et al. (2026) present the Teaching Monster Challenge, the first instructional-video generation benchmark that treats the learner persona as an explicit evaluation criterion, measuring whether AI agents can adapt a lesson to a specified learner — Pedagogical Content Knowledge (PCK). Systems receive a topic and a learner persona and must generate a complete instructional video, screened by an Large Language Models (LLMs)-judge, ranked by crowd pairwise voting, and finalized by an expert panel. The first edition shows systems handle content well but are far weaker at presenting and adapting it to the learner. It also exposes a limit of automatic judging: the LLM-judge separates a clear low-performing tail but ranks the strongest systems poorly and nearly identically, so its ranking does not match human preference.

Key Findings

1. Current systems are mature at delivering accurate, coherently organized content but far weaker at presenting it clearly and adapting it to the specified learner.
2. Learner adaptation is perceptible: human raters identify the intended learner persona of a video well above chance, so the adaptation PCK requires is present, only not yet done well.
3. The automated LLM-judge separates a clearly weak tail but scores the strongest systems near its ceiling with nearly identical scores, so its ranking of the top systems does not match human preference (Spearman ρ = −0.17).
4. Only the challenge winner beat the organizer baselines, so most systems have not yet clearly improved on a mature commercial product or a well-chosen human-made video.

Why Teaching Is Harder Than Generating Content

Teaching requires more than answering correctly. Following Shulman, teacher knowledge has three parts: content knowledge — mastery of the subject itself; pedagogical knowledge — the general craft of teaching that holds across subjects; and Pedagogical Content Knowledge (PCK), the blend that transforms a subject into a form a particular learner can grasp. Knowing a subject does not guarantee being able to make a learner understand it, so an agent's ability to teach depends on whether it can adapt what it knows to a given learner. The Teaching Monster Challenge is the first instructional video generation Benchmark to treat the learner persona as an explicit evaluation criterion, filling a gap left by existing benchmarks that measure Problem Solving accuracy, text-based tutor responses, or perceptual video quality but never adaptation to a specific learner.

Task Design

Framed as a single-shot task, each system receives a course requirement and a learner persona and must produce one complete instructional video end to end with no human in the loop. Asking for a complete video forces the system to break the concept down, order it, illustrate it, and deliver it across text, visuals, and narration — a full exercise in Learning Design. Items span four secondary-level STEM subjects (Physics, Biology, Computer Science, and Mathematics) anchored to the AP curriculum, a widely used secondary-school curriculum standard. A subset of items forms matched pairs that hold the course requirement fixed and vary the learner persona, so any remaining difference in a system's two videos is attributable to the manipulated learner — controlled evidence of adaptive teaching.

Method and Evaluation Framework

Because measuring student learning gains directly from each video is infeasible at scale, the benchmark evaluates teaching quality as a proxy, following established practice where observer ratings of teaching predict achievement. Evaluation runs through three layers. Layer 1 scores every submitted video with an automated LLM-judge across four dimensions that map onto PCK: Content Accuracy (subject-matter knowledge), Pedagogical Logic (pedagogical knowledge), Learner Adaptability (the adaptation at the core of PCK), and Engagement and Multimodal Presentation (drawing on the cognitive theory of multimedia learning). Layer 2 runs an Arena-style blind crowdsourced pairwise comparison aggregated into Elo ratings, and Layer 3 has an expert panel of teachers, principals, and professors rank the finalists directly. Only the top systems advance at each layer, concentrating the costly human evaluation on the strongest candidates. Across the warm-up and preliminary phases, 77 teams submitted 1,612 preliminary videos; the top ten advanced and three reached the final.

Systems and Baselines

Nearly all submissions instantiate a six-stage cascade: persona parsing and routing, lesson planning, slide and script composition, visual rendering, narration synthesis, and assembly. Recurring choices include structuring the persona as JSON conditioning for every stage, difficulty-based model routing, web-search grounding of lesson plans to reduce hallucination, Bloom's taxonomy constraints enforced by closed-loop LLM review, split handling of conceptual versus mathematical figures, and forced alignment of narration to animation. Three organizer-built baselines bound the design space: a cascaded pipeline, a commercial black-box generator (NotebookLM), and a static retrieval framework that reuses existing Creative Commons videos.

Findings

Systems handle content well but deliver and adapt it poorly. The two content dimensions — Accuracy and Logic — score higher than the two delivery dimensions — Engagement and Adaptability — and Adaptability is lowest overall. Grouping the 6,699 deduction flags shows visual delivery draws the largest share (39%), learner adaptation 27% (mostly missing Scaffolding, jargon overload, and prerequisite gaps), content problems only 16%, and narration 8%; about one video in six still contains a critical factual error, so high content scores coexist with occasional explanation mistakes.

A persona-identification study answers whether systems truly adapt: raters who watch a video and pick its intended learner among three candidates identify it well above the 33% chance baseline for system output, and near the topline for human-made videos — so the adaptation PCK requires is present, only not yet done well.

The LLM-judge separates clearly weak submissions but ranks the strongest poorly. All ten shortlisted systems score near the ceiling of 5, leaving little room to tell them apart, so its ranking of them has almost no agreement with the crowd ranking (Spearman ρ = −0.17): tsunumon, the eventual winner, ranked only fourth under the judge, while the judge's top two finished eighth and tenth with the crowd. This mismatch is not noise on either side, and it stems from a prevalence-versus-salience difference: the judge records criteria that apply to nearly every or almost no video (so they separate nothing), while the crowd's pairwise choices turn on the small differences their written rationales actually cite. The judge and human layers are therefore complementary — one screens every video, the other compares the strongest few.

What this means for practice

• Designers. Invest development effort in learner modeling and Scaffolding rather than factual coverage: adaptability was the lowest-scoring dimension, and learner adaptation accounted for 27% of deduction flags against only 16% for content problems.
• Designers. Condition every stage of the generation cascade on the parsed learner persona, not just the topic — the matched-persona pairs show that a difference in output is attributable to the learner only when the persona is carried through planning, script, visuals, and narration.
• Designers. Treat the pedagogical agent as the design target rather than a content pipeline, since instructional video generation is itself a generative teaching act and about one video in six still contained a critical factual error.
• Researchers. Keep a human comparison layer on top of automated judging: the LLM-judge scored all ten shortlisted systems near its ceiling and disagreed almost entirely with crowd preference (Spearman ρ = −0.17), ranking the eventual winner only fourth.
• Researchers. Release items, rubric, scoring traces, and human judgments as a single testbed so that progress on teaching systems and on the judges of teaching can advance together.

Limitations

• The benchmark covers only AP-aligned secondary STEM subjects — Physics, Biology, Computer Science, and Mathematics — with all videos generated in English; the authors state the findings may not generalize to other subject areas or non-English settings.
• Human evaluation reflects one education system: the final expert panel consisted mainly of secondary-school teachers, school principals, and university professors experienced in Taiwan's system, so its judgments track that system's instructional traditions rather than a universal standard.
• It evaluates single-shot instructional video generation, not interactive tutoring; systems do not adapt to later learner questions, errors, or confusion, leaving continuous diagnosis and individualized feedback untested.
• Student learning gains are never measured directly — teaching-quality ratings and the automated judge stand in as a proxy — and only the challenge winner beat the organizer baselines, so most systems had not clearly improved on a commercial product or a well-chosen human-made video.

Citation

Lin, Y.-C., Guo, Y.-K., Chen, S.-C., Feng, B.-H., Hsu, Y.-M., Hsieh, H., Lin, Y.-J., Wu, Y.-L., Dong, J.-K., Cheng, A.-Y., Huang, Y.-H., Ieong, L.-L., Chen, K.-Y., Tchouang, M.-D., Sun, S.-H., Lin, C., Ding, J.-J., & Lee, H.-y. (2026). Findings of the first Teaching Monster Challenge: A benchmark of pedagogical content knowledge in AI agents.