A review of intervention designs of LLM Integration in Undergraduate Computer Science Education

Synthesis: This scoping review analyzed 13 experimental studies on Large Language Models (LLMs) integration in undergraduate CS Education, examining how intervention design choices shape learning outcomes. The central finding: LLM effectiveness depends less on the model itself than on pedagogical design.

Three Intervention Archetypes

The "Tool Frustration" Paradox

A striking finding: students using Generative AI tools without adequate Scaffolding and prompt literacy training reported significantly higher frustration than controls (p = 0.008, median frustration 14 vs. 9), even when performance was equivalent. This mirrors concerns in the Over-Reliance and To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks literature — LLM access without pedagogical support can actively harm the learning experience.

Design Patterns That Work

The review identifies four design elements that distinguish effective interventions:

1. Sustained Scaffolding: Guided explanations, problem decomposition, and gradual reduction of support as competence grows — consistent with Vygotskian principles also discussed in Learning Design.
2. Transparent interaction patterns: Students need to understand how the LLM is reasoning, not just receive answers.
3. Explicit meta-skill instruction: Prompt Engineering literacy must be taught — students cannot intuit effective prompting strategies.
4. Assessment redesign: Emphasize code evaluation and prompt crafting over code generation, as also recommended in Reshaping Undergraduate Computer Science Education in the Generative AI Era.

Language and Methodological Gaps

Java interventions showed more consistent gains; Python — despite dominance in CS1 — lacks sufficient experimental isolation. The review also documents critical methodological weaknesses: inconsistent outcome operationalization, variable control group definitions, and chronic underreporting of effect sizes and confidence intervals — a concern that connects to broader efficacy-study design standards.

What this means for practice

• Instructors. Run LLM integration as a semester-long virtual tutor or peer with scaffolded feedback rather than a short-term coding assistant: all three tutor studies improved computational thinking and academic performance, while the eight task-and-coding-assistant studies split across measures.
• Instructors. Teach prompt literacy and supply Scaffolding before granting tool access, since students using LLMs without them reported significantly higher frustration than controls (p = 0.008, median 14 vs. 9) at equivalent performance — integration that skips that support is worse than no integration at all.
• Instructional designers. Build interventions on the four elements the review isolates: sustained scaffolding with gradual withdrawal, transparent interaction patterns, explicit meta-skill instruction in Prompt Engineering, and assessment redesigned around code evaluation and prompt crafting.
• Researchers. Pre-specify outcomes, control conditions, and effect sizes with confidence intervals in any new trial, because the reviewed studies operationalize outcomes inconsistently, define control groups differently, and chronically underreport effect sizes and intervals.

Limitations

• Only 13 experimental studies survived screening from 179 candidates (148 unique after deduplication), so each intervention archetype rests on a handful of studies — 3 for virtual tutors, 2 for exam and quiz help.
• The included studies underreport effect sizes and confidence intervals, which blocks pooled or meta-analytic effect estimation.
• Control conditions vary across studies (no internet access, no LLM access, historical data), so "control group" is not comparable from one study to the next.
• Programming-language effects are not isolated: Java studies showed more consistent gains while Python, despite its dominance in CS1, lacks sufficient experimental data.

Citation

Vissapragada, A. (2026). A review of intervention designs of LLM Integration in Undergraduate Computer Science Education. EdArXiv preprint.