CoLearn: An Agentic Tutor that Learns its Learner in a Human-AI Co-Learning Loop

Synthesis: CoLearn argues that a wrong answer is diagnostic content rather than a single bit, and builds an agentic tutor around that premise. A persistent learner-state memory stores per-topic mastery and mined misconceptions, updated by a soft-evidence variant of Bayesian Knowledge Tracing in which an Large Language Models (LLMs) grader acts as a continuous observation function. That memory drives question generation toward the learner's weakest topic, and the loop is exposed through a dashboard and a built-in blind A/B test so the personalization can be inspected rather than trusted. In blind comparison, adaptive questions were preferred over random-topic ones 68% of the time in Biology and 69% in Chemistry, and the agent's mastery estimates converged toward hidden ground truth.

Key Findings

1. Personalized questions won the blind comparison. Across 18 raters completing 12 comparisons each, adaptive items were preferred 0.68 of the time in Biology and 0.69 in Chemistry, against a 0.5 chance rate (two-sided sign test, p = 0.008).
2. Targeting collapses without a mastery memory. Adaptive sessions served items on a genuinely weak skill 0.72 of the time, against 0.55 (random-topic) and 0.57 (frozen) for the controls.
3. Belief error shrank under the adaptive condition. Final mastery mean absolute error was 0.12 under Adaptive versus 0.16 (Random-topic) and 0.20 (Frozen), and mean update magnitude decayed from 0.14 to 0.10 as estimates locked on.
4. Grading evidence tracked ability tiers but not individuals. Mean evidence rose cleanly across tiers (0.25 weak / 0.67 mixed / 0.77 strong), but within-tier correlation with true mastery was only r ≈ 0.15 / 0.48 / 0.41 — weakest, and worst, for low-ability learners.
5. Misconception mining is imperfect. Mined misconception labels matched personas' assigned misconceptions at F1 ≈ 0.56 across 132 rounds, so the diagnostic layer recovers roughly half the labeled misconceptions it targets.
6. A short-answer round costs about 32 seconds and $0.015. Multiple-choice rounds are graded deterministically with no LLM call and run about four times faster at 7.6 seconds and roughly $0.005 per round on the live single-node deployment.
7. Learners rated it usable. 16 raters scored 13 Likert items at a mean of 4.0 out of 5, no dimension below ≈3.7, and 12 of 16 (75%) would reuse it; responsiveness rated lowest.

A memory that is written, not just read

The control condition is what makes this design legible. CoLearn runs three session modes: Adaptive uses the full system, Random-topic draws items from randomly chosen in-scope topics while still updating memory, and Frozen serves fixed items in order while reading memory without writing it. Random-topic holds adaptivity constant and varies only targeting; Frozen varies only whether the memory accumulates. That is a cleaner decomposition than the usual adaptive-versus-not comparison, and the results follow it: the adaptive condition won on weak-skill targeting (0.72 vs 0.55 and 0.57) and on final mastery error (0.12 vs 0.16 and 0.20), while the two controls stayed flat. The memory itself is stored in per-(learner, subject) records rather than per-session logs, which is what lets evidence accumulate across sessions and keeps learner data local to the deployment (Privacy).

Where the observation function breaks down

CoLearn's diagnostic layer is an LLM grader emitting continuous mastery evidence and a confidence weight, blended into a confidence-shrunk posterior and passed through a competence-gated learning transition: a clearly wrong answer cannot raise the mastery estimate. The paper is unusually direct about the ceiling here: pooled per-answer evidence correlates with true mastery at r = 0.68, but that figure is driven mostly by between-tier separation, and within-tier correlation falls to r ≈ 0.15 for weak learners. Misconception recovery sits at F1 ≈ 0.56. The authors state plainly that the observation function is least reliable for exactly the learners who most need accurate misconception diagnosis, which makes grading fidelity the framework's binding constraint rather than its question strategy.

Making adaptation inspectable

The contribution distinguishing CoLearn from a conventional Adaptive Learning prototype is that its personalization is designed to be auditable. Learners see mastery bars animate after each answer and a "why this question" chip naming what the item was chosen to probe; the evidence view adds a branch-graph timeline, a concept-mastery radar and a mastery-over-time trajectory. The blind A/B tool presents adaptive and random-topic questions in randomized unlabelled slots and reports a preference rate with a confidence interval and a sign test. The paper also separates what the system does today from what it is designed for: the question-strategy guideline is fixed text consulted on every generation, and automatic revision of that strategy is planned, not implemented, so the live demo shows memory-side co-learning only.

What this means for practice

• Instructors. Treat a wrong answer as a label, not a score: tagging each distractor with the misconception it embodies turns a response into a diagnostic signal at no LLM cost.
• Faculty developers. Ask for the frozen-memory control when evaluating an adaptive tutor; CoLearn's targeting advantage falls from 0.72 to 0.57 once the memory stops accumulating.
• Developers. Budget for grading: short-answer rounds cost about 32 seconds and $0.015 each, while deterministic multiple-choice grading is roughly four times faster at 7.6 seconds.
• Researchers. Report within-tier reliability alongside pooled correlation — CoLearn's pooled r = 0.68 falls to r ≈ 0.15 for low-ability learners.

Limitations

• The mastery estimate is the agent's belief about the learner, not a measurement of knowledge; the paper makes no Learning Gains claim, and its A/B study measures perceived relevance rather than learning.
• BKT parameters are shared priors rather than fitted per skill, so absolute mastery values are indicative and not calibrated.
• The observation function is weakest for low-performing learners (within-tier r ≈ 0.15 for the weak tier versus 0.48 and 0.41 for mixed and strong), and persona evaluation used 6 synthetic learners over 132 rounds, with no far-transfer measure.
• Strategy self-evolution is planned but not implemented, and no direct comparison with other LLM tutors is made because of differences in experimental setup.

Connected Concepts

• Knowledge Tracing
• Learner Modeling and Adaptive Instruction
• Adaptive Learning
• Misconceptions about AI
• Personalized Learning
• Intelligent Tutoring
• Automated Question Generation
• Formative Assessment
• Cognitive Diagnosis
• Mastery Learning
• Large Language Models (LLMs)
• Simulating Students
• Explainable AI
• Learning Analytics
• Human AI Collaboration

Connected Articles

• Automated Recommendation of Programming Learning Content Using Pattern-based Knowledge Components — Automated Recommendation of Programming Learning Content Using Pattern-based Knowledge Components
• Adaptive Scaffolding for Cognitive Engagement in an Intelligent Tutoring System — Adaptive Scaffolding for Cognitive Engagement in an Intelligent Tutoring System
• The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions — The Correct Answer Trap: Pedagogically-Grounded Detection and Feedback for Hidden Misconceptions
• Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior — Curiosity as Linguistic Intervention: Using LLM Tutoring Dialogues to Influence Exploratory Learning Behavior
• Effectiveness of adaptive versus non-adaptive intelligent tutoring systems in early primary mathematics — Effectiveness of adaptive versus non-adaptive intelligent tutoring systems in early primary mathematics
• Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing — Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing
• CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming — CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming
• How much selection would be enough? Bounding the learning claim of El Salvador's artificial intelligence tutoring pilot — How much selection would be enough? Bounding the learning claim of El Salvador's artificial intelligence tutoring pilot

Citation

He, K., Wu, Z., Zhang, L., Zhao, R., He, Y., & Li, J. (2026). CoLearn: An Agentic Tutor that Learns its Learner in a Human-AI Co-Learning Loop. arXiv:2609.21154.