A review of intervention designs of LLM Integration in Undergraduate Computer Science Education

Key Findings

This scoping review analyzed 13 experimental studies on LLM integration in undergraduate CS Education, examining how intervention design choices shape learning outcomes. The central finding: LLM effectiveness depends less on the model itself than on pedagogical design.

Three Intervention Archetypes

The "Tool Frustration" Paradox

A striking finding: students using Generative AI tools without adequate Scaffolding and prompt literacy training reported significantly higher frustration than controls (p = 0.008, median frustration 14 vs. 9), even when performance was equivalent. This mirrors concerns in the Over Reliance and Critical Engagement Code Completion literature — LLM access without pedagogical support can actively harm the learning experience.

Design Patterns That Work

The review identifies four design elements that distinguish effective interventions:

1. Sustained Scaffolding: Guided explanations, problem decomposition, and gradual reduction of support as competence grows — consistent with Vygotskian principles also discussed in Instructional Design.

2. Transparent interaction patterns: Students need to understand how the LLM is reasoning, not just receive answers.

3. Explicit meta-skill instruction: Prompt Engineering literacy must be taught — students cannot intuit effective prompting strategies.

4. Assessment redesign: Emphasize code evaluation and prompt crafting over code generation, as also recommended in Reshaping CS Education GenAI.

Language and Methodological Gaps

Java interventions showed more consistent gains; Python — despite dominance in CS1 — lacks sufficient experimental isolation. The review also documents critical methodological weaknesses: inconsistent outcome operationalization, variable control group definitions, and chronic underreporting of effect sizes and confidence intervals — a concern that connects to broader efficacy-study design standards.

Relevance to AI in Education

This review is valuable because it shifts the conversation from "do LLMs work?" to "what design choices make LLMs effective?" The evidence strongly supports Scaffolding-based approaches over simple tool access, reinforcing findings across the GenAI Meta Analysis Programming Learning literature. The "tool frustration" paradox is an important contribution — it suggests that poorly designed Generative AI integration can be worse than no integration at all.

For Higher Ed contexts, the review provides actionable guidance: semester-long Virtual Tutor designs with structured feedback outperform short-term coding-assistant interventions. This aligns with Code Review GenAI Cs1 work on structured feedback and the broader CS Education push toward Computational Thinking over syntax mastery.

Connected Concepts

Connected Articles

Citation

Vissapragada, A. (2026). A review of intervention designs of LLM Integration in Undergraduate Computer Science Education. EdArXiv preprint.