AI Web Agents Can Effectively Guide Lesson Design and Predict Student Outcomes

Synthesis: Wang, Mitchell, and Piech (2025) show that a single Multimodal AI LLM-based web agent can evaluate an online learning experience before real students engage with it — autonomously navigating a multi-step lesson like a learner, generating a rich description of the student experience, and using that description to predict where students will drop out and to give designers actionable feedback. Validated on Code in Place, an open-access global CS1 course (6,515 students, 149 countries), agent-generated descriptions significantly improved dropout-distribution prediction (mean JSD 0.060), beating every baseline and ablations that lacked them. Notably, the authors found that a single describing agent outperformed simulating an entire distribution of simulated student agents — simulated students reproduced only ~4% of real student paths and were far more expensive, a cautionary result for the field.

Key Findings

1. A two-phase "describe, don't simulate" agent beats multi-agent student Simulation. The authors propose a single-agent pipeline: in phase one a web agent (built on WebVoyager) steps through the lesson — screenshotting pages, reading interactive elements, and parsing audio transcripts — producing a comprehensive description of the student experience (what is presented, what prior knowledge is required, how content is delivered, what the UI looks like, and where students might struggle). In phase two that description is fed to an LLM to predict outcomes and give design feedback. Directly simulating a distribution of students with persona-conditioned agents was both substantially more expensive and less informative — across 100 agents on five test lessons, simulated students covered only about 4% of the paths real students took, offering little insight into lesson difficulty.
2. Agent descriptions dramatically improve dropout prediction. Across 11 lessons in Code in Place, the agent-based method achieved an average Jensen–Shannon Divergence of 0.060 ± 0.003 between predicted and true dropout distributions — significantly lower (p = 0.000) than every other approach. Baselines scored 0.114 (sample) and 0.176 (regression); an LLM given only target-lesson details scored 0.116, adding previous-lesson details changed little (0.115), and adding real transfer data improved it to 0.100. Removing any single question from the agent's prompt nearly doubled the error, underscoring the value of each descriptive dimension.
3. Dropout prediction is the clear win; completion-rate prediction is mixed. For completion rates, agent descriptions offered a modest improvement for LLM methods (0.055 → 0.051), but a polynomial regression baseline achieved the lowest error overall (0.048), suggesting the agents do not capture all the factors driving whether students finish a lesson.
4. Case studies show the agent can compare lesson designs and flag confusion. Across four deliberately altered lesson versions, the agent (with an o1 comparator) preferred keeping a for-loops lecture video for novices, adopted shorter videos plus interactive fill-in-the-code exercises over long videos, recommended an always-visible lesson outline with progress indicators, and — strikingly — only succeeded in completing a Karel exercise when Karel was depicted as an arrow (whose directionality was most intuitive) rather than a robot or turtle. This demonstrates the agent can identify confusing content and explain the pedagogical impact of design choices.
5. A new Benchmark framing. The paper poses the Zero-Shot Learning Experience Evaluation Challenge: predict student outcomes and extract actionable design insights without testing on real students. It frames completion rate and the dropout distribution (where along the lesson students who start but don't finish are most likely to leave) as the two falsifiable, quantifiable validation metrics.

What this means for practice

• Instructors. Run an LLM web agent through a lesson before launch and treat its dropout distribution as a design check: agent-generated descriptions reduced prediction error to a mean Jensen–Shannon Divergence of 0.060 ± 0.003, against 0.114 for sampling real student behavior and 0.176 for regression on lesson features.
• Instructors. Ask the agent to compare alternative versions when you are weighing a change, not just to bless one design. In the case studies it preferred keeping the for-loops lecture video for novices, endorsed replacing a single long terminal video with four shorter videos plus two interactive fill-in-the-code examples, and recommended an always-visible lesson outline with progress indicators.
• Designers. Keep every descriptive dimension in the agent prompt and protect them in later edits: removing any single question from the agent's prompt nearly doubled the dropout-prediction error.
• Designers. Skip persona-based simulated student populations as an evaluation method. Across 100 agents on five test lessons they covered only about 4% of the paths real students took, cost substantially more compute, and produced less usable insight into lesson difficulty than one describing agent.
• Designers. Read the interface as pedagogy, not decoration: the agent completed the Karel lesson only when Karel was drawn as an arrow rather than a robot or turtle, showing how much a representation's perceived affordance shapes whether learners finish a lesson.

Limitations

• Validation is confined to one open-access CS1 course (Code in Place, 6,515 students, 149 countries): dropout prediction covers 11 lessons and completion-rate prediction 14, and the four case studies are deliberately altered versions of that same course's lessons, so no other subject, platform, or age group is tested.
• Completion-rate prediction did not beat a polynomial regression baseline (0.048 error, versus 0.051 for the best LLM method with agent descriptions), so the agent descriptions do not capture everything that determines whether students finish a lesson.
• The authors name bias in the LLM's training data and in the transfer data as the most significant limitation of the method, warning that if those data skew toward particular student behaviors the approach may favor some student groups while neglecting others.
• Technical limits remain: the model's context window and the requirement that the web agent be able to navigate the user interface, which the authors expect model and agent advances to lift rather than treat as solved.

Connected Concepts

• Learning Design — the web agent evaluates a learning experience holistically, blending pedagogy with UI interaction, before students engage
• Agentic AI — autonomous, goal-directed web agents that navigate and act on interfaces rather than only answering prompts
• Simulating Students — the paper's central finding that a single describing agent outperforms simulated student populations
• Learner Modeling and Adaptive Instruction — predicting dropout and completion as outcome modeling from agent-generated descriptions
• CS Education — the Code in Place CS1 validation context; Karel, control flow, terminal, expressions lessons
• Online Teaching and Learning — massive open-access online course evaluation at scale
• Generative AI — GPT-4o (agent) and o1 (comparator/predictor) multimodal LLMs
• Large Language Models (LLMs) — the underlying large language models
• AI Ed Evaluation — using AI to evaluate learning experiences and give design feedback without human testers
• Pedagogical Agent — webVoyager-based agent that behaves like a student navigating the interface

Connected Articles

• AgentSchool: An LLM-Powered Multi-Agent Simulation for Education — multi-agent simulation of education, contrasted with the single-agent finding here
• Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators — simulating students or sycophantic problem solving; epistemic fidelity versus the ~4% path coverage reported here
• Towards Valid Student Simulation with Large Language Models — validity criteria for simulated students; the competence paradox relates to why simulated agents underperform describing agents
• A Machine Learning Approach for Predicting Student Progress in Online Programming Education — predicting dropout and progress in large-scale online programming courses
• ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents — benchmarking LLM agents for instructional design; complementary to using agents to evaluate designs
• From MOOC to MAIC: Reshaping Online Teaching and Learning through LLM-driven Agents — AI agents reshaping MOOC learning, the "N agents for 1 student" inversion versus "one video for N students"
• Generative AI technologies and educational outcomes: a comprehensive meta-analysis comparing traditional and AI-driven approaches — predicting educational outcomes with generative AI

Citation

Wang, S., Mitchell, J., & Piech, C. (2025). AI Web Agents Can Effectively Guide Lesson Design and Predict Student Outcomes. In Artificial Intelligence in Education (AIED 2025), pp. 290–304. Springer.