Cao, Zhao, Schunn, McLaughlin, Lin & Koedinger (2026) — UNC Chapel Hill, CMU, U Pittsburgh, U Hong Kong. Randomized experiment (n=199).
Core Finding
Sequenced AI feedback harms learning despite boosting engagement and positive perceptions.
In a randomized experiment with 199 participants, the authors compared two types of AI-generated feedback:
- Sequenced (layered): Encouragement → hints → correct answer, designed to promote learner autonomy
- Non-sequenced (direct): Full-solution feedback immediately
Contrary to design intuition, sequenced feedback led to significantly poorer learning performance. The finding reveals a critical disconnect between what students like and what actually helps them learn.
Mediation Pathways
Three causal pathways were tested via mediation analysis:
| Pathway | Mediator | Effect | Significant? |
|---|---|---|---|
| Affective | Perceived encouragement | Positive → better learning | ✓ |
| Behavioral | Tasks needing ≥3 submissions | Negative → worse learning | ✓ |
| Cognitive | Mental effort | Neutral | ✗ |
The positive affective pathway (students felt more encouraged) was completely counteracted by the negative behavioral pathway (students made more resubmissions). The net effect was significantly poorer learning outcomes.
Key Mechanisms
Why Sequenced Feedback Backfired
- The hint-before-answer structure inadvertently encouraged trial-and-error behavior rather than deep processing
- Students submitted more attempts per task, indicating they were "gaming" the hint system rather than engaging in genuine problem-solving
- Higher mental effort was reported but did not translate to better learning — suggesting the effort was directed at navigating the feedback sequence rather than understanding the material
Why Direct Feedback Worked
- Immediate corrective information eliminated the temptation to guess
- Students processed the solution rather than iterating through hints
- Lower engagement scores but higher learning outcomes
Design Implications
This study challenges the prevailing intuition that more scaffolded, autonomy-supportive feedback is always better. Key takeaways for AI feedback system design:
1. Engagement ≠ learning: User satisfaction and behavioral engagement are not reliable proxies for learning gains — designers must measure learning outcomes directly 2. Limit resubmission loops: Systems should cap hint requests or require reflection between attempts to prevent trial-and-error gaming 3. Strategic blending: Consider providing direct corrective feedback first, with optional encouragement and hints available on demand rather than as a mandatory sequence 4. Cognitive load management: The higher mental effort induced by sequenced feedback did not aid learning — design should channel effort toward understanding rather than navigation
Connection to Existing Wiki
This paper directly informs several threads in the wiki:
- formative-assessment: Direct evidence about AI-generated feedback design — sequencing that feels supportive may undermine formative goals
- critical-thinking-genai-scaffolding: Vendrell & Johnston's eight design principles for LLM scaffolding — this study provides empirical evidence that poorly designed scaffolding can harm learning, reinforcing the need for "cognitive friction" design
- prober-ai-inquiry-writing: The inverted paradigm (AI asks questions, gates suggestions) offers an alternative to sequenced feedback that may avoid the resubmission trap
- self-regulated-learning: Sequenced feedback was intended to promote autonomy and SRL, but the behavioral data shows it had the opposite effect — a cautionary tale for SRL-aligned AI design
- metacognition: The engagement-learning disconnect exemplifies the metacognitive calibration problem — students felt they were learning more with sequenced feedback when they were actually learning less
- ai-peer-feedback-systems: Multi-LLM collaborative feedback systems must consider feedback sequencing carefully to avoid the pitfalls identified here
- pedagogy-ai-mistakes: Hosseini's work on deliberately leveraging AI errors connects to the finding that easy, encouraging feedback may be less pedagogically effective than direct correction
- ai-learning-transfer: The learning outcome disparity between conditions raises transfer implications — do sequenced-feedback students retain less when the scaffolding is removed?
Methodological Strengths
- Randomized controlled design with 199 participants — causal claims are well-supported
- Mediation analysis identifies why the effect occurs, not just whether it occurs
- Multi-dimensional measurement: learning performance, behavioral engagement (submission patterns), cognitive engagement (mental effort), and affective perceptions
- Multi-institution collaboration (UNC, CMU, Pitt, HKU)
Open Questions
- Would results differ with longer exposure (multi-session vs. single-session study)?
- Does domain matter — would sequenced feedback work better for ill-defined problems than well-defined ones?
- Can the resubmission problem be solved by requiring reflection prompts between hint levels?
- Would a hybrid design (direct feedback + optional hints) preserve learning while maintaining positive affect?
Related Pages
- understanding-student-effort-response-time — Understanding Student Effort Using Response-Time Propensitie...
- self-referential-l2-writing-llm-assessment — Identifies which feedback dimensions benefit from AI vs. human input
- short-answer-scoring-quality-degradation — Binary scoring at extremes undermines nuanced feedback for learning
- multimodal-ai-feedback-learning — Zhao et al.: direct AI multimodal feedback matches educators; contrasts with Cao et al. finding that sequenced feedback harms learning
- automated-grading — Automated feedback and grading systems
- ai-tutor-effectiveness-review — Broader ITS effectiveness evidence
- generate-then-validate-question-gen — Reducing hallucination in AI-generated educational content
- human-in-the-loop-ai — Strategic interleaving of AI generation with human judgment
- engagement-assessment-video — Measuring engagement in learning contexts
Citation
APA: Cao, J., Zhao, C. Q., Schunn, C., McLaughlin, E. A., Lin, J., & Koedinger, K. R. (2026). Assessing the Impact and Underlying Pathways of Sequenced AI Feedback on Student Learning. arXiv:2604.07469.