A meta-analysis of the effect of generative AI on productivity and learning in programming

Synthesis: Maier, Gunzenhäuser & Schweisthal (2026) conduct a meta-analysis synthesizing evidence on how generative AI tools affect both programming productivity and learning outcomes. This is a confidence: high paper due to its synthesis design across multiple studies, addressing the central tension between short-term efficiency gains and long-term skill development.

Core Contribution

Maier, Gunzenhäuser & Schweisthal (2026) conduct a meta-analysis synthesizing evidence on how generative AI tools affect both programming productivity and learning outcomes. This is a confidence: high paper due to its synthesis design across multiple studies, addressing the central tension between short-term efficiency gains and long-term skill development.

Key Findings

The meta-analysis examines the productivity-learning trade-off that sits at the heart of Transfer of Learning debates: when AI tools boost immediate coding output, do they simultaneously undermine the development of foundational programming skills? This directly connects to the broader question of whether The Evidence Base on AI in K-12: A 2026 Review matters — general AI coding assistants may optimize for productivity at the expense of learning.

Significance for AIED

This paper provides the highest-level evidence synthesis currently available on the impact of generative AI in programming education. It bridges RCT-level findings with practical implications for:

• Curriculum design: When should AI tools be introduced in programming courses?
• Assessment: How do we validly assess programming skill when AI is available?
• Policy: What guidance should AI Regulation in Education frameworks provide for AI in CS education?

The meta-analytic approach provides more robust conclusions than individual studies like ProPACT: A Proactive AI-Driven Adaptive Collaborative Tutor for Pair Programming (ProPACT) or Agentic Education with AI Coding Assistants work on teaching Claude Code. It also connects to Programming Intelligent Tutoring Systems research on traditional intelligent tutoring for coding, providing a baseline for comparing AI-augmented approaches.

The findings have implications beyond programming — the productivity-learning tension applies to Writing, Language Learning, and any domain where AI can produce acceptable output without the learner developing underlying skills (the The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows problem).

Open Questions

• Does the productivity-learning trade-off vary by student skill level (novices vs. experts)?
• How do findings generalize beyond programming to other STEM Education domains?
• What instructional designs mitigate the learning cost while preserving productivity gains?

What this means for practice

• Instructors. Treat a faster programming lab as a throughput gain, not a learning gain: GenAI assistance produced a moderate, statistically significant productivity effect (Hedges' g = 0.33, 95% CI: [0.09, 0.58]) while the pooled learning effect was g = 0.14 with a confidence interval that includes zero.
• Curriculum designers. Decide deliberately whether students keep GenAI access during assessment. The learning evidence comes from exam performance comparisons and the effect depended strongly on contextual setting, including whether students retained access to GenAI during the test, so the access rule is a design decision, not a detail.
• Educators. Do not port findings from a controlled trial into your own course unexamined: productivity gains were larger in controlled experimental settings with standardized tasks and smaller in real-world contexts, and heterogeneity across studies was extreme (I² = 99%).
• Curriculum designers. Choose the skill you intend to build before adding a coding assistant, because the productivity proxies used in these studies (task completion time, commits, and lines of output code) measure output produced, not programming competence retained.
• Researchers. Replicate with bias checks built in: the pre-registered review searched ACM, arXiv, Scopus and Web of Science for 2019–2025, yet Egger's mixed-effects regression test indicated significant funnel plot asymmetry, so effect sizes in this literature are likely inflated.

Limitations

• The learning conclusion rests on a small base — 11 effect size estimates, against n = 23 studies and k = 27 effect sizes overall — and is not distinguishable from zero (Hedges' g = 0.14, 95% CI: [−0.18, 0.47], SE = 0.17, p = 0.389), with between-study variance of τ² = 0.25 (Q(10) = 54.96, p < 0.001).
• Productivity is measured by proxies rather than by quality: the 16 effect sizes from n = 14 studies (m = 3535 participants; r = 6355 repositories) used task completion time, commits, and lines of output code, and only one study used lines of output code at all.
• Publication bias is present: Egger's mixed-effects regression test indicated significant funnel plot asymmetry and small-study effects, so the productivity estimate is likely optimistic.
• The productivity pool is dominated by one tool and a narrow set of languages: GitHub Copilot in n = 6 studies, with Java the most frequent language (n = 6), leaving other assistants and languages thinly evidenced.

Citation

Maier, S., Gunzenhäuser, M., & Schweisthal, J. (2026). A meta-analysis of the effect of generative AI on productivity and learning in programming.