🧠 AI Ed Wiki

📄 Research Article

Assessing the Impact and Underlying Pathways of Sequenced AI Feedback on Student Learning

● high · 2026-05-11feedback-loop, formative-assessment, scaffolding, generative-ai, efficacy-study, student-experience, self-regulated-learning, automated-grading, human-in-the-loop-ai, metacognition

Sequenced AI Feedback on Student Learning

Core Finding

Sequenced AI feedback harms learning despite boosting engagement and positive perceptions.

In a randomized experiment with 199 participants, the authors compared two types of AI-generated feedback:

  • Sequenced (layered): Encouragement → hints → correct answer, designed to promote learner autonomy
  • Non-sequenced (direct): Full-solution feedback immediately

    • Contrary to design intuition, sequenced feedback led to significantly poorer learning performance. The finding reveals a critical disconnect between what students like and what actually helps them learn.

    Mediation Pathways

    Three causal pathways were tested via mediation analysis:

    PathwayMediatorEffectSignificant?
    AffectivePerceived encouragementPositive → better learning✓
    BehavioralTasks needing ≥3 submissionsNegative → worse learning✓
    CognitiveMental effortNeutral✗

    The positive affective pathway (students felt more encouraged) was completely counteracted by the negative behavioral pathway (students made more resubmissions). The net effect was significantly poorer learning outcomes.

    Key Mechanisms

    Why Sequenced Feedback Backfired

  • The hint-before-answer structure inadvertently encouraged trial-and-error behavior rather than deep processing
  • Students submitted more attempts per task, indicating they were "gaming" the hint system rather than engaging in genuine problem-solving
  • Higher mental effort was reported but did not translate to better learning — suggesting the effort was directed at navigating the feedback sequence rather than understanding the material

    • Why Direct Feedback Worked

  • Immediate corrective information eliminated the temptation to guess
  • Students processed the solution rather than iterating through hints
  • Lower engagement scores but higher learning outcomes

    • Design Implications

    This study challenges the prevailing intuition that more scaffolded, autonomy-supportive feedback is always better. Key takeaways for AI feedback system design:

    1. Engagement ≠ learning: User satisfaction and behavioral engagement are not reliable proxies for learning gains — designers must measure learning outcomes directly

    2. Limit resubmission loops: Systems should cap hint requests or require reflection between attempts to prevent trial-and-error gaming

    3. Strategic blending: Consider providing direct corrective feedback first, with optional encouragement and hints available on demand rather than as a mandatory sequence

    4. Cognitive load management: The higher mental effort induced by sequenced feedback did not aid learning — design should channel effort toward understanding rather than navigation

    Connection to Existing Wiki

    This paper directly informs several threads in the wiki:

  • Formative Assessment: Direct evidence about AI-generated feedback design — sequencing that feels supportive may undermine formative goals
  • Critical Thinking GenAI Scaffolding: Vendrell & Johnston's eight design principles for LLM scaffolding — this study provides empirical evidence that poorly designed scaffolding can harm learning, reinforcing the need for "cognitive friction" design
  • Prober AI Inquiry Writing: The inverted paradigm (AI asks questions, gates suggestions) offers an alternative to sequenced feedback that may avoid the resubmission trap
  • Self Regulated Learning: Sequenced feedback was intended to promote autonomy and SRL, but the behavioral data shows it had the opposite effect — a cautionary tale for SRL-aligned AI design
  • Metacognition: The engagement-learning disconnect exemplifies the metacognitive calibration problem — students felt they were learning more with sequenced feedback when they were actually learning less
  • AI Peer Feedback Systems: Multi-LLM collaborative feedback systems must consider feedback sequencing carefully to avoid the pitfalls identified here
  • Pedagogy AI Mistakes: Hosseini's work on deliberately leveraging AI errors connects to the finding that easy, encouraging feedback may be less pedagogically effective than direct correction
  • Transfer Of Learning: The learning outcome disparity between conditions raises transfer implications — do sequenced-feedback students retain less when the scaffolding is removed?

    • Methodological Strengths

  • Randomized controlled design with 199 participants — causal claims are well-supported
  • Mediation analysis identifies why the effect occurs, not just whether it occurs
  • Multi-dimensional measurement: learning performance, behavioral engagement (submission patterns), cognitive engagement (mental effort), and affective perceptions
  • Multi-institution collaboration (UNC, CMU, Pitt, HKU)

    • Open Questions

  • Would results differ with longer exposure (multi-session vs. single-session study)?
  • Does domain matter — would sequenced feedback work better for ill-defined problems than well-defined ones?
  • Can the resubmission problem be solved by requiring reflection prompts between hint levels?
  • Would a hybrid design (direct feedback + optional hints) preserve learning while maintaining positive affect?

    • Connected Concepts

  • Formative Assessment
  • Self Regulated Learning
  • Metacognition

    • Connected Articles

  • Critical Thinking GenAI Scaffolding
  • Prober AI Inquiry Writing
  • AI Peer Feedback Systems
  • Pedagogy AI Mistakes
  • Transfer Of Learning

    • Citation

    Cao, J., Zhao, C. Q., Schunn, C., McLaughlin, E. A., Lin, J., & Koedinger, K. R. (2026). Assessing the Impact and Underlying Pathways of Sequenced AI Feedback on Student Learning. arXiv:2604.07469.