LLM-Generated Summaries for Teachers: A Randomized Field Experiment in a Digital Learning Platform

Synthesis: Lim, Worden, Sales and Heffernan ran a pre-registered randomized field experiment on ASSISTments, a digital learning platform used in K-12 mathematics, to test whether LLM-generated assignment summaries reduce the cost of reading reports. 874 teachers were randomized to business-as-usual reports or to reports carrying a "Generate Summary" feature, and about 40% of the treatment group used it at least once. Access produced no statistically significant change in logged teacher behavior, yet students of treated teachers started and completed more assignments: about 3–4 percentage points under intent-to-treat, with +7.7pp and +9.1pp among teachers who used them. The authors interpret the split as teacher follow-up happening off the platform, and argue that scaling such support requires room in the workflow, not only better summaries.

Key Findings

1. Access alone did not change logged platform behavior. Pre-registered intent-to-treat estimates for creating and viewing reports were small and positive but not statistically significant (874 teachers in the first analytic sample, 810 in the second), and the local average treatment effects scaled by use were also non-significant. The authors call the confidence intervals wide relative to the estimates.
2. Students started more assignments. The average proportion of students who started assignments was significantly higher among teachers in the summary condition, with covariate-adjusted intent-to-treat gains of about 3–4 percentage points.
3. Completions rose by a similar margin. The average proportion of students who completed assignments was also significantly higher, again about 3–4 percentage points under intent-to-treat with covariate adjustment.
4. Use was associated with larger gains than access. Two-stage least squares estimates put the effect of actually using summaries at +7.7pp for assignment starts and +9.1pp for completions.
5. Most teachers never used it. 38% of the treatment group in the first analytic sample and 40% in the second generated a summary at least once (first-stage F = 268.5 and 276.4). Of the 440 treatment teachers, 62% never generated one, 48% of those who tried it generated only a single summary, and 10 teachers accounted for more than ten each.
6. Teachers with fewer assignments gained more. A post-hoc exploratory analysis found larger gains among teachers who created fewer assignments, which the authors suggest may reflect an easier route from insight to action when the assignment set is smaller.
7. Survey ratings were positive but thinly supported. Teachers who responded to follow-up surveys generally rated the summaries positively, though response rates were low.

What the study did

The study starts from an observation about practice: teachers are familiar with digital learning platforms, but many still struggle to use assignment reports. Formative reports can support instructional decisions, yet limited time and data literacy make them feel overwhelming. The intervention is a design response to that overload rather than a new assessment instrument. Instead of asking teachers to work through the report themselves, the platform offers a short, plain-language summary of the key insights for each assignment, generated by a large language model, with the explicit aim of lowering the cognitive load of interpreting the report.

The trial ran on ASSISTments, a platform used in K-12 mathematics. Assignment was at the teacher level. One arm received the standard assignment reports; the other received the same reports plus access to a "Generate Summary" feature attached to each assignment. The design is one-way noncompliant by construction: only teachers with access could use summaries, and many of those never did, so the experiment compares an offer rather than a treatment that everyone in the condition received.

That structure is why the paper reports two estimands. The intent-to-treat effect answers what happens when a platform makes summaries available, which is the decision a vendor or district actually makes. The local average treatment effect answers what happens when a teacher uses them, estimated by two-stage least squares with random assignment as the instrument for use. The second number is larger than the first in part because it is scaled by the roughly 40% who took up the offer. Both sets of analyses were pre-registered, and the authors report data, code and an appendix alongside the pre-registration.

Outcomes came in three families. Platform logs captured teacher behaviors such as creating assignments and viewing reports. Student records captured the average proportion of students who started and who completed assignments. Follow-up surveys captured teachers' own ratings of the summaries. The student outcomes are the ones that moved.

Why the effect appears off the platform

The pattern is unusual: the feature changed student behavior without changing the teacher behaviors the platform records. The authors' reading is that summaries shifted what teachers did after they closed the report. A teacher who reads that a class missed a particular skill may regroup students, reteach the item the next morning, or adjust the next assignment — actions that leave no trace in the platform's event log but do show up when students open and finish the next assignment.

The paper frames the underlying problem as two separate costs. The first is "time-to-understand," the effort of working out what a report is saying; summarization is aimed squarely at that cost, and it is where model quality and personalization matter. The second is "time-to-act," the effort of turning an insight into a changed plan. A clearer summary does nothing about the second cost if the teacher has no slot in the day for review and no routine that connects a report to the next instructional move. This distinction explains why a feature that plausibly succeeded at comprehension produced a null on the logged behaviors closest to comprehension.

The post-hoc finding about assignment volume is consistent with that account. Teachers who created fewer assignments had a smaller set of results to act on, so the distance between reading a summary and changing something was shorter. The authors are careful to label this exploratory, and it should be read that way: it is a hypothesis about the mechanism, generated from the same data rather than tested against it.

The take-up rate also shapes how the results should be read. With about 40% of the treatment group using the feature at least once, the intent-to-treat estimate averages an effect over many teachers who never saw a summary. The local average treatment effect is the better guide to what a summary does when it is used, but it is identified only under the instrumental-variable assumptions that access affects outcomes solely through use, which the design invokes rather than verifies.

What this means for practice

• Instructors. Give report review a standing slot rather than treating it as something to get to when the week allows, because the trial's student-side gains appeared without any corresponding rise in report viewing, which suggests the constraint sits after comprehension rather than in it.
• Administrators. Evaluate summary features on assignment starts and completions rather than on clicks on the summary itself, and expect a substantial share of licensed teachers not to use the feature at all in its first term.
• Designers. Put the summary next to the action it implies — the next assignment, the regrouping decision, the reteach list — so that reading it and acting on it are one step, since the evidence points at time-to-act as the binding constraint.
• Researchers. Pre-register platform-log and downstream outcomes together and report intent-to-treat alongside local average treatment effects, because a moderate take-up rate can make an access-only analysis look null while per-user effects are considerably larger.
• Policymakers. Cost these tools as workflow interventions with an adoption curve, not as information interventions whose value is realized the moment they are switched on.

Limitations

The trial covers one platform, subject and level: ASSISTments in K-12 mathematics, with teachers randomized. The student outcomes are assignment starts and completions, which are engagement traces rather than measures of learning, so it cannot show that students learned more — only that they did more of the assigned work. The null estimates on logged teacher behavior come with wide confidence intervals relative to their size, so the absence of a significant effect should not be read as proof that nothing changed. Student participation measures are teacher-level aggregates averaged across assignments, which is why the authors ran an assignment-level check after removing the highest-volume quartile. The subgroup finding on assignment volume is post-hoc. Survey evidence rests on a low response rate, which makes the positive ratings weak evidence on perceptions. And the design cannot observe the offline follow-up actions that the authors invoke to explain the pattern, so that explanation remains an interpretation rather than a measured mechanism.

Citation

Lim, W.-C. I., Worden, E., Sales, A., & Heffernan, N. T., III. (2026). LLM-Generated Summaries for Teachers: A Randomized Field Experiment in a Digital Learning Platform. In Proceedings of the Thirteenth ACM Conference on Learning @ Scale (L@S '26). ACM. DOI: 10.1145/3774398.3811606.