AI Cosmos: An Agentic Infrastructure for Situated Learning

Synthesis: AI Cosmos (2026) starts from a familiar worry stated precisely: gains made with a Large Language Models (LLMs) during a task do not show that the learner can do the task afterwards. Its answer is an infrastructure that structures practice as a scenario — a learning objective, role-specialized agents, queryable knowledge, and a stateful workbench — and preserves the residue of that practice as typed, provenance-bearing assets that later scenarios can retrieve. Two preregistered three-arm randomized trials test the claim: legal reasoning with 131 students and competitive programming with 54 completers. Against a vanilla assistant the system produced a 7.0-point larger learning gain (d = 0.56) in legal reasoning, concentrated on objective scenario items rather than open argument, and a larger share of students reaching criterion mastery in programming. The mechanism the authors emphasize is structural: students took about 1.7× as many turns and wrote roughly 40% fewer characters per turn, so the effect is not explained by talking more.

Key Findings

1. The design separates practice from the artifact. A scenario is defined by an objective, participant roles, knowledge resources and workbench operations; an asset is a typed unit — role specification, scenario specification, knowledge artifact, process record or experience trace — that keeps its provenance and its dependencies on earlier assets.
2. Legal reasoning improved against a plain chatbot. Across 131 students in three arms, AI Cosmos gained 22.8 percentage points against 15.7 for the vanilla assistant, a 7.0-point advantage (Welch t(71.8) = 2.42, p = 0.018, d = 0.56).
3. A teacher persona alone did not help. LLM-as-Teacher gained 2.8 points over vanilla, p = 0.121, and trailed AI Cosmos by 4.2 points, p = 0.146, so prompting the same model to behave like a tutor produced no detectable benefit.
4. The gain sat in structured items, not in argument. On objective scenario-based items AI Cosmos exceeded vanilla by 8.3 points, 95% CI [1.8, 14.9], p = 0.018, while open-response gains of 5.6 to 7.8 points did not differ across arms (p = 0.80).
5. The interaction looked different, not just longer. Total student input was indistinguishable across arms, yet AI Cosmos elicited 51.0 turns on average against 29.6 and 29.2 (F(2, 108) = 21.8, p = 1.1 × 10−8), and about 40% fewer characters per turn (162 versus 259 and 276).
6. Programming mastery separated where test-case scores did not. In the second trial, criterion mastery at grade G4 or better reached 68% under AI Cosmos against 45% for vanilla and 32% for LLM-as-Teacher, even though successful online-judge submissions were comparable at 82% and 85%.
7. Retention was directionally consistent and inconclusive. At four weeks the ordering held (+18.9 points retained for AI Cosmos against +14.7 for vanilla), but with 18 participants per arm the comparison was underpowered, F(2, 51) = 0.48, p = 0.62.

The scenario and the asset

The agentic framework is best read as a data-model argument. A scenario binds the social, epistemic and operational parts of practice: which agents and learners participate and in what roles, what knowledge can be queried, and what workbench the learner acts through with persistent state. A programming task is therefore not a chat about programming but a session in an IDE with an objective, an explainer and a mentor.

Assets are the second half. A process record keeps the sequence of actions, critiques and revisions that produced an outcome, and an experience trace keeps structured evidence such as failure cases and observed relationships between configuration choices and performance. Because dependencies are recorded, the memory forms a directed acyclic graph, and later scenarios retrieve prior practice rather than rediscovering it. That is what separates it from an ITS with a chat window: the system is situated rather than merely conversational.

What the trials actually tested

Both studies are three-arm, preregistered and randomized, with the same base model held constant across arms, which is the design's strongest feature: the comparison isolates the infrastructure from the model. The legal-reasoning trial assigned 131 students in a 1:1:1 ratio and used ANCOVA on post-test with pre-test as covariate, with the planned contrast against vanilla reported alongside an omnibus test that only approached significance (p = 0.054).

The programming trial is the more interesting one methodologically. Conventional online-judge scores give little diagnostic information about a partial implementation, so two expert problem authors rescored all submitted code blind to condition on a five-level mastery scale (G1–G5). Expert scores correlated with judge scores at r = 0.92 to 0.96 yet separated the arms where binary test outcomes did not, which is the paper's clearest demonstration that the outcome measure determines the finding.

Where the evidence stops

Two effects in the paper are large and one is a near-miss. The legal-reasoning advantage over the vanilla assistant is reliable and medium-sized, and it survives adjustment for baseline. The advantage over the stronger teacher-persona arm is positive but not significant, and that arm itself did not beat vanilla, so the honest summary is that structure beat a plain assistant, not that it beat tutoring.

The durability result should be read as a question left open. Eighteen participants per condition cannot detect a 4-point difference, and the authors say so. The retention claim in the abstract is directional; the data support "did not decay detectably" rather than "retained".

What this means for practice

• Instructional designers. Treat the workbench as the curriculum. The measurable difference here came from giving students a stateful environment to act in and an objective to act toward, not from better explanations, which the teacher-persona arm showed to be inert.
• Educational technology developers. Keep the process record. The asset model's value is that a later task inherits the conditions and revisions of an earlier one, which requires recording provenance at the moment of practice rather than reconstructing it.
• Instructors. Expect more turns and shorter messages. The interaction signature here was iterative and terse; a student writing 40% fewer characters per turn but twice as many turns is working differently, not less.
• Assessment designers. Choose the measure before the trial. The programming result appeared only under expert mastery scoring, which suggests that binary correctness will understate structured-practice effects.

Limitations

• The legal-reasoning omnibus test did not reach significance (F(2, 108) = 2.94, p = 0.057); the claim rests on the preregistered planned contrast, and the advantage over LLM-as-Teacher (Δ = 4.2, p = 0.146) is not significant.
• The programming sample is 54 completers from 140 recruited, 93% male, mean age 18.9, so attrition and composition limit generalization.
• Subgroup and item-type analyses were exploratory and unadjusted for multiplicity, as the authors state.
• The delayed test had 18 participants per arm, making the retention comparison underpowered by the authors' own account.
• Both studies evaluate one system built by its authors against baselines of their choosing, and the assets' cross-scenario retrieval is not itself measured.

Citation

AI Cosmos Team, Zhou, J., Chen, Y., Chang, F., Su, M., Zhu, Y., Wang, Y., … Li, M. (2026). AI Cosmos: An Agentic Infra for Situated Learning. arXiv:2610.40031.