🧠 AI Ed Wiki

📄 Research Article

A meta-analysis of the effect of generative AI on productivity and learning in programming

● high · 2026-05-06rct, efficacy-study, generative-ai, higher-ed, learning-gains, meta-analysis, regulation, writing-education, assessment-validity, llm

Maier, Gunzenhäuser & Schweisthal (2026) conduct a meta-analysis synthesizing evidence on how generative AI tools affect both programming productivity and learning outcomes. This is a confidence: high paper due to its synthesis design across multiple studies, addressing the central tension between short-term efficiency gains and long-term skill development.

Generative AI Meta-Analysis: Programming Productivity vs. Learning

Core Contribution

Maier, Gunzenhäuser & Schweisthal (2026) conduct a meta-analysis synthesizing evidence on how generative AI tools affect both programming productivity and learning outcomes. This is a confidence: high paper due to its synthesis design across multiple studies, addressing the central tension between short-term efficiency gains and long-term skill development.

Key Findings

The meta-analysis examines the productivity-learning trade-off that sits at the heart of Transfer Of Learning debates: when AI tools boost immediate coding output, do they simultaneously undermine the development of foundational programming skills? This directly connects to the broader question of whether Tutoring Specific Vs General AI matters — general AI coding assistants may optimize for productivity at the expense of learning.

Significance for AIED

This paper provides the highest-level evidence synthesis currently available on the impact of generative AI in programming education. It bridges RCT-level findings with practical implications for:

  • Curriculum design: When should AI tools be introduced in programming courses?
  • Assessment: How do we validly assess programming skill when AI is available?
  • Policy: What guidance should Regulation frameworks provide for AI in CS education?

    • The meta-analytic approach provides more robust conclusions than individual studies like Collaborative AI Tutoring (ProPACT) or Agentic Education Coding work on teaching Claude Code. It also connects to Programming ITS research on traditional intelligent tutoring for coding, providing a baseline for comparing AI-augmented approaches.

    The findings have implications beyond programming — the productivity-learning tension applies to Writing Education, Language Learning, and any domain where AI can produce acceptable output without the learner developing underlying skills (the LLM Fallacy Misattribution problem).

    Open Questions

  • Does the productivity-learning trade-off vary by student skill level (novices vs. experts)?
  • How do findings generalize beyond programming to other STEM Education domains?
  • What instructional designs mitigate the learning cost while preserving productivity gains?

    • Connected Concepts

  • RCT
  • Assessment Validity
  • Regulation
  • Writing Education
  • Language Learning
  • STEM Education

    • Connected Articles

  • Transfer Of Learning
  • Tutoring Specific Vs General AI
  • Collaborative AI Tutoring
  • Agentic Education Coding
  • Programming ITS
  • LLM Fallacy Misattribution

    • Citation

    Schweisthal, A.S.M.M.G.J., on, A.M.O.T.E.O.G.A., Manuel, S.M.M.G.J.S., 1,2, S.A.S.F., Munich, L.M., Munich, M.C.F.M.L., searched, O.G.C.A.O.P.A.L.W.S., & us-, A.S.W.R.T.C.G.W.U.P. (2026). A meta-analysis of the effect of generative AI on productivity and learning in programming. contrast, we find no statistically significant effect of GenAI assistance on learning outcomes (g = 0