AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study

Synthesis: Multi-institutional study on Generated Animated Traces (GATs) for CS1. Found that mid-engagement students may experience a performance decrement due to coordination costs (Expertise-Reversal Effect). CS Education, Scaffolding, Personalized Learning, STEM Education, Adaptive Learning.

Key Findings

• Generated Animated Traces (GATs) — AI-generated, analogy-based, narrated animations that coordinate source code, execution state, and conceptual analogies — were evaluated against textual explanations in CS1 courses at two institutions (Python, N=961; Java, N=151).
• GATs yielded selective benefits for immediate learning, but the benefits were context-dependent and short-term rather than uniformly positive.
• GATs' influence on performance was moderated by learners' engagement profiles: mid-engagement students could experience a performance decrement, interpreted in terms of coordination costs consistent with the expertise-reversal effect.
• End-of-course engagement and exam performance did not show that GATs produced durable advantages over textual explanations.
• The findings underscore the importance of personalizing instructional support to learner characteristics rather than treating AI-generated visualizations as universally beneficial.

Study Design & Method

The multi-institutional study compared GATs with textual explanations in introductory programming courses at two universities, one teaching Python and one teaching Java. Immediate learning performance and learner experience were measured at the point of instruction, followed by end-of-course engagement and exam performance to test longer-term effects. Learner engagement profiles were derived from the data and used as moderators, allowing the authors to detect differential effects that aggregate analyses could mask. The framing draws on program-Visualization research and cognitive load theory, in which the effectiveness of visualizations depends on design and context.

What this means for practice

• Instructors. Do not adopt AI-generated animated traces (GATs) uniformly across a CS1 course: the benefits for immediate learning were selective and did not carry through to final-exam performance, so treat them as one tool in the instructional palette rather than a replacement for textual explanations.
• Instructors. Target GATs by learner state rather than topic alone — mid-engagement students showed a performance decrement consistent with the expertise-reversal effect, which argues for personalized or adaptive deployment instead of whole-course rollout.
• Instructors. Choose the modality deliberately even when materials are matched: GATs and textual explanations were equated on objectives, scope, and line-by-line execution order, yet effects stayed context-dependent across the Python (N = 961) and Java (N = 151) deployments.
• Designers. Design AI-generated visualizations to cut extraneous load rather than add it: the proposed mechanism for the mid-engagement decrement was the coordination cost of tracking code, execution state, and analogy at once.
• Researchers. Plan for sustained integration if durable gains are the goal, since advantages measured right after the intervention had not transferred to exam performance by the end of the course.

Limitations

• Two non-comparable deployments: the Python and Java courses differed in programming language, participation incentives, and topic coverage, so the authors do not pool the data or draw cross-institutional conclusions.
• Narrow topic coverage and no long-term transfer: interventions addressed a limited set of topics, and the immediate effects did not transfer to final-exam performance.
• The moderation finding is exploratory and self-reported: engagement profiles came from k-means clustering on self-report CAP instruments, and cognitive load, frustration, and situational interest were Likert items, so the learner-difference results are suggestive rather than confirmatory.

Citation

Noviello et al. (2026). AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study.