Presumed Effective: The Manufacturing of an Evidence Base for AI-in-Education Through Flawed Meta-Analysis

Synthesis: O'Neill (2026) subjects 14 peer-reviewed meta-analyses that claim AI, GenAI, or ChatGPT improves education to a forensic methodological audit, and finds that none provided a valid basis for the claims it advanced. The most basic failures are conceptual: all but two of the meta-analyses define the treatment as a tool rather than a pedagogical intervention, and multidimensional outcomes are pooled as though their dimensions were interchangeable. An evidentiary audit of 59 vetted primary studies found that 61% of the randomly selected studies were problematic, most often because the outcome measured did not match the meta-analysis claim. Reported heterogeneity was high in every meta-analysis that provided I², no moderator analysis met the minimum subgroup size, and no meta-analysis offered a valid and sufficient accounting of publication bias. O'Neill argues these failures point to failed editorial and reviewer gatekeeping rather than isolated errors, and recommends mandatory data transparency for meta-analyses together with stronger peer review and retraction practice.

Key Findings

1. Audit design. The study is a forensic methodological audit of 14 peer-reviewed meta-analyses, published after 1 January 2023, whose abstracts claim that AI or a specific AI technology affected education, academic performance, academic achievement, learning outcomes, learning performance, or higher-order thinking. It combines structured document analysis of those meta-analyses, an evidentiary audit of 59 primary studies (46 randomly selected plus 13 targeted for a priori suspicion), and a targeted citation-uptake analysis of two of the meta-analyses.
2. No coherent construct. All but two meta-analyses defined the treatment as a tool such as ChatGPT or GenAI, and a product name is not a pedagogy: treating exposure to ChatGPT as a common intervention is comparable to meta-analyzing the effects of "paper." Outcomes such as academic achievement bundled test scores, homework quality, motivation, self-efficacy, attitudes, and engagement into a single pooled estimate, and 11 of the 14 meta-analyses set no population boundary, so "students" spanned children through medical trainees.
3. Primary-study vetting. Of the 46 randomly selected primary studies, 28 (61%) presented validity concerns, and all 13 targeted studies were problematic. The most common concerns were dependent-variable mismatch (n = 15), independent-variable mismatch (n = 11), experimental design problems (n = 7), data extraction problems (n = 6), absence of a control group (n = 6), and nonrandom group assignment (n = 6).
4. Contaminated studies passed quality screening. Alneyadi and Wardat (2023) cite six articles about ChatGPT dated before ChatGPT existed, attach an incorrect DOI, and report η² = .0403 as "greater than 0.14"; it was included by six of the meta-analyses, including four of the five that applied formal primary-study quality metrics. The companion Alneyadi and Wardat (2024), included by three meta-analyses, reports η² = 1.374, which is impossible because η² cannot exceed 1.
5. Statistical independence was widely violated. Twelve of the 14 meta-analyses treated multiple effect sizes drawn from the same primary study as independent, for example describing "a total of 86 independent studies in 22 publications" (Doo & Park, 2026) or "65 independent studies" (Sun & Zhou, 2024). This inflates the apparent amount of evidence, underestimates standard errors, narrows confidence intervals, distorts the pooled mean, and makes significance and moderator effects easier to obtain.
6. Heterogeneity was high and left unresolved. Reported heterogeneity was high in every meta-analysis that supplied I² (range 77.2% to 94.4%), and 12 of the 13 reported values exceeded 80%. Only four meta-analyses reported global τ², and only two reported prediction intervals, both of which included zero. Because I² is precision-dependent, O'Neill notes that these statistics do not by themselves indicate the absolute dispersion of true effects, and the missing τ² and prediction intervals are what leave the pooled means uninterpretable.
7. Moderator analysis was underpowered and confounded. No meta-analysis met the baseline recommendation that the smallest subgroup include at least ten studies; five included subgroups with only a single study and three others had subgroups as small as two studies, which cannot meaningfully explain heterogeneity. Twelve violated the independence assumptions of their own moderator tests, none attempted to minimize confounding among correlated characteristics, and eight evaluated more than five moderator variables (some as many as 13) without any adjustment to the critical value for Type I error.
8. Publication bias was never validly assessed. All 14 meta-analyses attempted an assessment, but none provided a valid and sufficient accounting. Egger's test was used by 10 and Begg's test alone by one; 10 of those 11 tests violated independence assumptions. The critical value was always set at p < .05, and applying the recommended p < .10 would have classified five additional tests as significant. Discredited fail-safe N metrics were used in six meta-analyses and were the only quantification attempt in three of them, and negative low-power tests were reported as proof that no bias existed. Only Liu et al. (2025) made valid calculations assessing publication bias.
9. A cascade of errors produced a concrete policy recommendation. Liu et al. (2025) and Sun and Zhou (2024) treated study sample size as class size and mis-extracted the Ji et al. (2023) primary study, whose tables list numbers of students (N = 36) while the group is the unit of analysis (N = 8). The miscalculated effect size was placed in the wrong sample-size subgroup and drove the shared finding that 21 to 40 students is the ideal intervention size for ChatGPT/GenAI, on which basis both meta-analyses recommended that interventions be designed for that number of students.
10. Flawed evidence circulates as authority. The audited meta-analyses appeared in journals published by Springer Nature, Wiley, SAGE, Taylor & Francis, MDPI, and Athabasca University, and had accumulated 2,467 combined citations by 25 July 2026. Wang and Fan (2025) circulated for 351 days and accrued more than 500 citations before being retracted on 22 April 2026; of 20 post-retraction citing papers sampled, 12 (60%) cited it as authoritative evidence that AI improves learning and none acknowledged the retraction. Lee and Kwon (2024) was misread as evidence that AI integration improves learning in 8 of 14 sampled citing papers (57%), with only 2 (14%) citing it appropriately.
11. The problem is systemic, not isolated author error. Elementary validity failures recurred across peer-reviewed journals from major publishers, including some of the highest impact venues in the field, which shifts the concern to the reliability of editorial screening, peer review, and journal standards for AIED meta-analysis. O'Neill recommends requiring full data transparency (search protocols, selection decisions, coded characteristics, extracted statistics, effect-size calculations, and analytic code) and restructuring review so that prior publication of a meta-analysis is not treated as evidence of reviewer competence and at least one reviewer vets a subset of primary studies, alongside COPE-compliant retraction practice with visible retraction notices and citation records.

What this means for practice

• Researchers. Before citing an AIED meta-analysis, check whether the treatment is a designed learning activity or a product name. All but two of the 14 audited meta-analyses defined the intervention as a tool such as ChatGPT or GenAI, which cannot carry a causal claim about learning — treating exposure to ChatGPT as a common intervention is, the author notes, comparable to meta-analyzing the effects of "paper."
• Researchers. Treat pooled estimates as unstable until the statistics support them: reported heterogeneity (I²) ranged from 77.2% to 94.4% across the 13 meta-analyses that supplied it, only two reported prediction intervals and both included zero, and 12 of the 14 meta-analyses counted multiple effect sizes from a single primary study as independent, which inflates the apparent evidence base and underestimates standard errors — and no meta-analysis met the ten-study minimum subgroup size for moderator analysis.
• Policymakers. Require independent methodological vetting before an AIED meta-analysis informs procurement, grant priority, or national AI-in-education policy: the audited corpus had accumulated 2,467 combined citations by 25 July 2026 and was published by Springer Nature, Wiley, SAGE, Taylor & Francis, MDPI, and Athabasca University. As the failures O'Neill documents point to failed editorial and reviewer gatekeeping rather than isolated author error, journals and editors should also require full data transparency for meta-analyses — search protocols, coded characteristics, extracted statistics, effect-size calculations, and analytic code — together with statistical review competence.
• Administrators. Add explicit retraction flags and notes to indexes, reading lists, and citation exports. Wang and Fan (2025) accrued more than 500 citations before being retracted on 22 April 2026, and of 20 post-retraction citing papers sampled, 12 (60%) cited it as authoritative evidence that AI improves learning while none acknowledged the retraction.
• Researchers. Ask reviewers to spot-check a sample of primary studies rather than trusting a meta-analysis's own coding. The evidentiary audit flagged 61% of the randomly selected primary studies (28 of 46), and the most common problems were dependent-variable mismatch (n = 15) and independent-variable mismatch (n = 11).

Limitations

• The audit assesses threshold-level validity only — whether meta-analyses met minimum evidentiary standards — and the author states that deeper educational validity failures (whether an effect size is educationally meaningful, durable, or scalable) were minimally assessed.
• Only part of the primary-study sample was selected randomly: 5 studies were drawn at random from each of 12 of the 14 meta-analyses (60 selections representing 46 unique publications) and 13 more were targeted for a priori suspicion. The 61% prevalence figure (28/46) applies only to the random subset; targeted counts cannot be used as prevalence estimators.
• The corpus was not intended to be exhaustive: the study did not attempt to identify every AIED meta-analysis published during the review period, so the findings do not give the prevalence of validity failures across the field as a whole.
• Several audit thresholds are deliberately permissive operational screening criteria rather than assertions of safe practice — for example, flagging studies that tested more than five moderator variables without adjusting the critical value.

Citation

• O'Neill, P. E. (2026). Presumed Effective: The Manufacturing of an Evidence Base for AI-in-Education Through Flawed Meta-Analysis. EdArXiv preprint.