Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course

Synthesis: This study asks which students actually gain when a university course issues Generative AI study materials, and argues the answer depends on who bears the checking. In a compulsory first-year economics course, one half received AI-generated podcasts, FAQs, and quiz-based study guides produced with a source-grounded RAG (Retrieval-Augmented Generation) model and verified by a named graduate teaching assistant before release. Access was associated with a 2.34-mark advantage on a 50-mark examination component, but roughly three-quarters of that gain arose in the bottom quintile and the share of marks below the 60% classification boundary fell by 24.7 percentage points. The authors conclude that moving the checking burden from students to an accountable tutor lets verified materials reach the weaker students they are meant to help.

Key Findings

1. A two-cohort difference-in-differences design found that access to expert-verified AI study materials was associated with a 2.34-mark advantage on a 50-mark examination component (p = 0.045, HC1), about 0.33 to 0.40 standard deviations.
2. The share of marks below the upper-second classification boundary (30 of 50 marks, or 60%) fell by 24.7 percentage points relative to the counterfactual (p = 0.001), equivalent to 21 students in a cohort of 85.
3. Estimates were significant at every threshold from 23 to 31 marks and at none above 31; at the lower-second boundary (25 marks) the reduction was 12.9 points (p = 0.008).
4. Roughly three-quarters of the 2.34-mark average arose in the bottom quintile, whose difference-in-differences was 8.9 marks, with almost none originating in the top three quintiles.
5. The threshold estimate was robust to removing the lowest-scoring pre-intervention students (−19.9 to −22.9 points, p ≤ 0.006), while the average effect weakened from 2.34 to 1.41 marks when eight students were dropped.
6. Interviews and feedback from 36 students indicated the verification label gave them a reason to engage without ending their scrutiny; each podcast reached more than 70 unique listeners, about 82% of the cohort.
7. Producing and verifying each weekly study set took about 30 minutes, roughly 2.5 hours across five weeks, or about 1.8 minutes of staff time per student.

The judgment burden

The authors argue that whether Generative AI narrows or widens attainment gaps depends on the judgment burden: the expertise and effort a learner must supply to separate usable from unusable AI output before learning from it. That burden has two parts — how variable the output is, and who is responsible for checking it. Screening an AI explanation against the course consumes time and working memory without advancing understanding, an extraneous load the authors relate to Cognitive Offloading while distinguishing their construct from cognitive load theory. Because checking draws on subject knowledge (Prior Knowledge), time, and confidence that weaker students have least of, material released unchecked should help stronger students most. Where a tutor checks it before release, the support reaches weaker students without the screening cost — verification relocates the burden rather than removing it. Equity therefore turns on a design choice institutional responses (Educational AI Policy) rarely name.

What the study did

The setting was a compulsory first-year Principles of Economics course at a UK business school (Business Education), with two five-week halves and a terminal examination (Summative Assessment) worth 50 marks per half. In 2023/24 neither half received materials. In 2024/25 the macroeconomics half received AI-generated podcasts, FAQs, and quiz-based study guides built with Google NotebookLM, a source-grounded Large Language Models (LLMs), from the course's own slides and readings, checked by a named graduate teaching assistant, and labeled "AI-generated and academically verified" (AI Use and Disclosure Statements). Materials were released free through the virtual learning environment, designed for multiple means of access in line with Universal Design for Learning, and mapped to the first-year audience. The quiz guides with model answers also invited self-testing with Feedback. The quantitative analysis (Quantitative Research) compared 170 students and 340 examination marks across the two cohorts.

Where the gains landed

Distributional estimators located the change. The treated half's standard deviation fell from 7.09 to 5.03 (Brown–Forsythe p = 0.004) while the untreated half's barely moved, and its interdecile range narrowed from 19.6 to 11.6 marks. At the upper-second boundary the treated share of marks below 30 fell from 23.5% to 10.6% as the untreated share rose from 7.1% to 18.8%, a difference-in-differences of −24.7 points. Quantile estimates were 11.0 marks at the 10th percentile (95% CI 5.0 to 13.0) and 5.0 at the 20th, with estimates near zero from the 30th upward. Decomposing the average (Learning Gains) attributed roughly three-quarters to the bottom quintile. The pattern is what the judgment-burden account predicts and bears directly on Differential Effects Across Learner Groups.

How students received verified material

Interviews with 36 students describe a consistent sequence: skepticism, reassurance from the label, then use. Students said the label gave them a reason to engage with material they would otherwise have doubted, and the Trust it conferred attached to a person they knew rather than to the model. Crucially, it did not end scrutiny — one student reported becoming "more critical, not less". Several described relief from the load of continually fact-checking AI output. The accounts most aligned with the quantitative pattern came from students facing constraints on conventional study: a second language (Multilingual Learning), long commutes, reluctance to ask questions, and paid work. These constraints limit the time, language, and confidence that screening consumes.

What this means for practice

• Instructors. Have a named tutor with subject expertise and a teaching relationship check AI-generated material before release, and regenerate vague or misaligned sections rather than approving first drafts.
• Administrators. Budget verification in workload allocations: it is the entire additional cost once generation is nearly free, and absorbing it into the unrecorded margins of insecure teaching contracts converts an efficiency gain into unpaid work.
• Instructional designers. Label both AI origin and verification, and name the verifier, since the disclosure itself prompted scrutiny; keep a log of verification time and changes per item so a decline in checking is visible before it reaches students.
• Researchers. Report distributional quantities — the change in the share of students below consequential grade boundaries, the change in dispersion, and the threshold or quantile profile — alongside any average effect (AI Ed Evaluation).

Limitations

• The evidence comes from one course at one institution with a single pre-intervention cohort, so parallel trends cannot be tested and cohort differences cannot be excluded.
• Effort substitution between the two examined halves remains possible; the component correlation moved against it (0.284 to 0.482) but not significantly (Fisher z = 1.50, p = 0.135).
• The average effect depends on a small number of students and loses significance when three to five are removed; only the threshold result is robust.
• Individual usage logs were not retained, so use could not be linked to outcomes, and the qualitative material consists of field notes rather than transcripts.
• The intervention bundled verification with new Multimodal AI material, so the contribution of verification relative to the material itself cannot be identified.
• The cohort was high-attaining with no failures (module means 70.0 and 69.1), leaving effects on failure and the pass boundary untested.

Citation

Dang, C. T., & Nguyen, A. (2026). Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course. arXiv:2610.07097.