AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study

Multi-institutional study on Generated Animated Traces (GATs) for CS1. Found that mid-engagement students may experience a performance decrement due to coordination costs (Expertise-Reversal Effect). CS Education, Scaffolding, Personalized Learning, STEM Education, Adaptive Learning.

Key Findings

Generated Animated Traces (GATs) — AI-generated, analogy-based, narrated animations that coordinate source code, execution state, and conceptual analogies — were evaluated against textual explanations in CS1 courses at two institutions (Python, N=961; Java, N=151).

GATs yielded selective benefits for immediate learning, but the benefits were context-dependent and short-term rather than uniformly positive.

GATs' influence on performance was moderated by learners' engagement profiles: mid-engagement students could experience a performance decrement, interpreted in terms of coordination costs consistent with the expertise-reversal effect.

End-of-course engagement and exam performance did not show that GATs produced durable advantages over textual explanations.

The findings underscore the importance of personalizing instructional support to learner characteristics rather than treating AI-generated visualizations as universally beneficial.

Study Design & Method

The multi-institutional study compared GATs with textual explanations in introductory programming courses at two universities, one teaching Python and one teaching Java. Immediate learning performance and learner experience were measured at the point of instruction, followed by end-of-course engagement and exam performance to test longer-term effects. Learner engagement profiles were derived from the data and used as moderators, allowing the authors to detect differential effects that aggregate analyses could mask. The framing draws on program-visualization research and cognitive load theory, in which the effectiveness of visualizations depends on design and context.

Implications for AI in Education

The study is a cautionary counterpoint to enthusiasm for AI-generated learning media: generative visualizations are not automatically better than text, and their effects vary by learner. For CS Education, the results argue for personalized or adaptive deployment of GATs — aligned with Personalized Learning and Adaptive Learning — rather than uniform adoption across a course. The expertise-reversal-style finding, where mid-engagement students bore coordination costs, suggests that scaffolding decisions must consider the learner's current state, and that AI-generated resources should be designed to reduce extraneous Cognitive Load Theory rather than add to it. Because benefits were short-term and context-dependent, GATs are best treated as one tool within a broader instructional palette rather than a replacement for established explanations.

Connected Concepts

CS Education

Scaffolding

Personalized Learning

STEM Education

Adaptive Learning

Connected Articles

Prompt Problems Nl Programming Mistakes — Understanding Student Perceptions, Mistakes, and Debugging Approaches when Solving Natural Language Programming Tasks

Automated Presentation Coaching — A Survey of Automated Presentation Coaching: Systems, Methods, and Open Challenges

Student Misconceptions Conditionals Loops Taxonomy — How Students (Mis)understand Conditionals and Loops -- A Taxonomy

Constructing Epistemic AI Literacy Student AI Co Programming — Constructing Epistemic AI Literacy: Detecting Epistemic Aims and Processes in Student-AI Co-Programming

Prompt Coach Agentic Tutor Prompt Engineering — Prompt Coach: An Empirical Evaluation of an Agentic Tutor for Learning Prompt Engineering in Software Development

LLM Misconception Difficulty Easy Trap — The Easy Trap: Why LLMs Underestimate Misconception-Driven Difficulty

Citation

Noviello et al. (2026). AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study. arXiv:2606.03288.