Is GenAI helpful for writing? A re-analysis using robust Bayesian meta-analysis (RoBMA)

Synthesis: Shawn Hemelstrand's short re-analysis note takes one headline number from a published meta-analysis and asks what survives when the estimation method changes. Dong (2026) reported a large effect of generative AI on writing, Hedge's g = 0.76 [0.18, 1.35], from only 13 studies, alongside strong heterogeneity, publication bias and an extreme outlier (one study at g = 3.752 [2.52, 4.98]); the original author applied trim-and-fill and leave-one-out as robustness checks, techniques the re-analysis notes "do not always help and can sometimes even harm the trustworthiness of results." Hemelstrand instead runs robust Bayesian meta-analysis (RoBMA), which averages over publication-bias models and priors rather than committing to one. The effect collapses: g = .018 [−0.377, 0.553], Bayes factor 0.320, while heterogeneity is near-certain (posterior probability 100%) and publication bias probable (98.6%). Removing the outlier weakens the bias evidence (BF = 1.74) but leaves heterogeneity high, with a weak effect indistinguishable from noise (g = 0.245 [−0.128, 0.87]). The note's contribution is methodological: a large claim about writing does not survive a method better suited to the conditions that produced it. Confidence in this page reflects faithful reporting of the source, not endorsement of its conclusion.

Key Findings

1. The original claim being re-examined. Dong's meta-analysis reported that generative AI has a large impact on writing outcomes, Hedge's g = 0.76 [0.18, 1.35] — characterized in this note as a "strong impact" and treated as the target of the re-analysis.
2. The original analysis was already flagged as fragile by its own results. It reported strong heterogeneity, publication bias, extreme outliers (one study at g = 3.752 [2.52, 4.98]) and only k = 13 studies. The original author's robustness techniques, trim-and-fill and leave-one-out meta-analysis, are described here as sometimes harming rather than helping trustworthiness.
3. Under RoBMA the average effect essentially disappears. The initial model put the effect of GenAI on writing at g = .018 [−0.377, 0.553] with a Bayes factor of 0.320 — "substantially lower" than the original estimate and evidence in favor of the null rather than the effect.
4. Heterogeneity was the dominant feature of the model. Posterior probability for heterogeneity reached 100% (BF > 49,999), and publication bias was also probable (PP = 98.6%, BF = 71.254). Evidence for selection bias specifically was very low, and model fitness checks (autocorrelation and trace plots) were acceptable.
5. The result hinged on a small-study-bias model and one outlier. The outcome appeared driven mainly by the PEESE model and the outlier study. With the outlier removed, evidence for publication bias weakened to BF = 1.74 and PEESE became less influential, yet heterogeneity stayed high and the effect was g = 0.245 [−0.128, 0.87], a weak effect indiscernible from statistical noise.
6. The priors were mostly RoBMA defaults, with one explicit informative prior. Defaults "assume bias and draw parameter estimates closer to zero." Only the average-effect parameter was set explicitly: μ ~ Normal(.40, 1), justified by Hattie's finding that educational intervention effects across more than 2,000 effect sizes usually center near d = .40. Estimation used 10,000 samples and 5 chains to raise the effective sample size.
7. The frequentist results were reproduced before the Bayesian re-analysis. The original frequentist analysis was reproduced with the metafor package using the author's publicly available data; the software or code used for the original analysis was not detailed in the original article. Conclusion stated plainly: after correcting for heterogeneity and small-study bias, the re-analysis "doesn't support the original claim of strong effects."

What question the note asks, and which analysis it re-examines

The note is a re-analysis rather than a new study. Its target is Y. Dong's "Generative AI technologies and educational outcomes: A comprehensive meta-analysis comparing traditional and AI-driven approaches," published in Humanities and Social Sciences Communications, which reported the g = 0.76 estimate for writing from 13 studies. The note is generous about the original author's intent — robustness techniques such as trim-and-fill and leave-one-out meta-analysis were employed "in good faith" — but argues these specific techniques are unreliable safeguards. The question is therefore narrow and consequential: given the same data, what does the effect of generative AI on writing outcomes look like when estimation is done with a method designed for the problems the data contain? For a knowledge base tracking what AI does to learning outcomes, the interest is less the single number than the demonstration that a widely cited effect size can be a property of the pipeline rather than the phenomenon.

What RoBMA adds over the original method

Robust Bayesian meta-analysis, implemented through the RoBMA R package (version 4.6.0), replaces the single-model decision that trim-and-fill and leave-one-out embody with model averaging. Rather than selecting one correction for publication bias — one selection model, one PET-PEESE specification — RoBMA averages across a family of models and priors, so the reported effect is a posterior distribution weighted by how well each account of the data performs. This matters precisely in the conditions Dong's analysis described: heterogeneity, small-study bias and a handful of studies, where picking one correction can determine the answer.

The prior structure is the second difference. RoBMA's defaults are deliberately conservative, assuming bias and pulling estimates toward zero, which means the analysis is not set up to flatter large effects. The one hand-set prior is the average effect itself, μ ~ Normal(.40, 1), anchored on Hattie's synthesis of educational intervention effects. Since the prior is informative and centered well below the original g = 0.76, the collapse to near zero cannot be attributed to an uninformative prior letting the likelihood drift; it is a genuine shift in what the data support. The note also points readers to Harrer et al. (2021) for the terminology, positioning the work as methodological translation rather than a technical novelty.

What this means for practice

• Researchers. Re-estimate a pooled effect with model averaging before treating it as settled. The same 13-study dataset behind g = 0.76 yields g = .018 [−0.377, 0.553] under RoBMA with Bayes factor 0.320, so state which publication-bias model and prior the headline number depends on.
• Researchers. Report the outlier-removed estimate beside the full-data one. Dropping the extreme study (g = 3.752) left g = 0.245 [−0.128, 0.87], an interval that still spans zero with heterogeneity near-certain.
• Researchers. Pre-specify and justify any informative average-effect prior. Here μ ~ Normal(.40, 1), anchored on Hattie's synthesis of more than 2,000 effect sizes, ensures the collapse toward zero cannot be attributed to an uninformative prior.
• Instructors. Treat any single pooled effect size for GenAI and writing as provisional when planning instruction: with 13 primary studies and heterogeneity at 100% posterior probability, one number summarizes a heterogeneous literature poorly in the original analysis and the re-analysis alike.
• Researchers. Post data and analysis code with meta-analyses. This re-analysis was feasible because the original study's data were public, and the note's own R code is posted on the same basis.

Limitations

• The re-analysis concerns one outcome, writing, from one meta-analysis of 13 studies, and supplies no new data — its argument is a re-estimation in the same family as other critiques of the AI-in-education evidence base — so it cannot show the true effect is zero, nor adjudicate the quality of those 13 primary studies.
• Both specifications stop short of a large, dependable effect: the full-data model gives a point estimate of .018 with a credible interval straddling zero and a Bayes factor below 1, and once the extreme outlier is removed the estimate roughly doubles to 0.245 with an interval of [−0.128, 0.87] that still includes zero, with heterogeneity remaining high.
• It is a single-author note with no DOI or venue in the source, whereas the original appeared in a peer-reviewed journal, so the asymmetry in scrutiny between the two is worth naming.
• The informative μ ~ Normal(.40, 1) prior is defensible and transparent but still a substantive choice: different priors would produce different posteriors.

Citation

Hemelstrand, S. (2026). Is GenAI helpful for writing? A re-analysis using robust Bayesian meta-analysis (RoBMA). KIMEP University, Department of Psychology. Data and R code: OSF project rnqzp.