Effect of Artificial Intelligence on Learning: A Meta-Meta-Analysis

Synthesis: Effect of Artificial Intelligence on Learning: A Meta-Meta-Analysis — A study-level meta-meta-analysis by Bartoš, Bujak, Martinková, and Wagenmakers (2026) that re-analyzes 1,840 effect sizes from 67 published meta-analyses of AI/Large Language Models (LLMs) effects on learning. Using publication-bias-adjusted robust Bayesian model-averaging, they find strong evidence of severe publication bias and extreme between-study heterogeneity, and estimate the bias-adjusted average effect at roughly one-third the magnitude reported in the literature (SMD = 0.196), with prediction intervals spanning large negative to large positive effects. They conclude that broad claims of generalized learning gains from AI/LLMs are premature and the evidence is insufficient to support policy or practice recommendations.

Key Findings

• Severe publication bias and extreme heterogeneity. Across 1,840 effect sizes from 67 meta-analyses, funnel plots show a strong right skew (all Egger tests p < .0001), indicating suppression of null/negative results, combined with extreme between-study heterogeneity (τ = 0.869).
• Bias-adjusted effects are ~one-third of reported estimates. Publication-bias-adjusted model-averaged effect was SMD = 0.196 [0.000, 0.323], versus the median unadjusted pooled effect of SMD = 0.67 that dominates the published literature. Published meta-analyses substantially overestimate the average effect.
• Wide prediction interval spanning harm to benefit. The true effect for a new study is predicted to fall between −1.521 and +1.908 standard deviations — spanning extreme negative to extreme positive outcomes. An effect that small and that variable cannot justify confident deployment decisions.
• No moderators yield consistent benefits. Subgroup analyses by outcome type (academic achievement, cognition, language, emotion, motivation), educational field/level, and AI role (tutor, partner, tool) failed to substantially reduce heterogeneity or identify subgroups with reliable gains.
• No difference pre- vs. post-ChatGPT. There was no substantial difference between studies published before vs. after January 2023, undercutting claims that modern LLM tools specifically produce gains.
• Heterogeneity means new studies add almost no information. An additional infinite-size study would add fewer than three observations' worth of effective sample size given the existing heterogeneity — so accumulating more studies of the current type will barely move estimates. The authors recommend high-quality, large-scale pre-registered studies and registered replication reports analyzed separately.
• Direct implications for meta-research quality. Reporting quality was poor enough that effect sizes could not be recovered from 7 of the meta-analyses. The results also align with re-analyses (Bardach et al.; Bartoš et al. 2025) showing that prominent positive meta-analyses like Wang & Fan (2025) — since retracted — had effects that disappear entirely under publication-bias adjustment.
• Caveat on scope. The estimates capture only short-term, direct effects on traditional learning measures; AI may also reshape education indirectly (Metacognition, instruction, feedback, assessment) in ways these metrics miss — so null/heterogeneous average effects may partly reflect measurement misalignment rather than zero educational value.

What this means for practice

• Researchers. Treat published pooled effects as upper bounds, not estimates: across 1,840 effect sizes from 67 meta-analyses the bias-adjusted model-averaged effect was SMD = 0.196, against a median published effect of SMD = 0.67.
• Researchers. Stop treating additional conventional studies as evidence accumulation — with between-study heterogeneity at τ = 0.869, an additional infinitely large study would add fewer than three observations' worth of effective sample size.
• Researchers. Fund and report pre-registered, large-scale studies or registered replication reports analyzed separately, because the prediction interval for a new study runs from −1.521 to +1.908 standard deviations.
• Policymakers. Do not cite the current meta-analytic literature as support for AI or LLM deployment rules; subgroup analyses by outcome type, educational field and level, and the role of AI found no group with consistent benefits.
• Researchers. Test the measurement, not only the delivery: these estimates cover short-term, direct effects on traditional learning measures, which can miss indirect change in Metacognition, instruction, Feedback, and Assessment.

Limitations

• The analysis is second-order: effect sizes and standard errors could not be recovered from seven of the published meta-analyses, and nine were excluded when re-computed pooled estimates diverged from the originals by more than SMD = 0.05.
• Of the 1,840 effect size estimates, 632 had no reported publication year, so the pre- versus post-ChatGPT comparison rests on 690 and 518 estimates respectively.
• The publication-bias adjustment is model-based — Bayesian model averaging over selection models and PET-PEESE — and the authors run a weight-function-only sensitivity analysis precisely because PET-PEESE can over-correct genuine small-study differences.
• The outcome space is narrow: short-term, direct effects on traditional learning measures, so nothing here speaks to indirect effects such as Metacognition, feedback, or assessment change.

Citation

Bartoš, F., Bujak, O. Z., Martinková, P., & Wagenmakers, E.-J. (2026). Effect of Artificial Intelligence on Learning: A Meta-Meta-Analysis. PsyArXiv Preprints.