Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI

Synthesis: An interpretable decision layer that ranks course topics needing instructor attention without using grades or post-hoc outcome labels. The system combines three process-level signals to identify which topics (and which students) need intervention before formal assessments.

What It Is

How It Works

Three signals feed into a topic priority score:

1. Gap prevalence (Rₜ): Fraction of students showing difficulty with topic t, detected from Jill Watson interaction traces
2. Survey disagreement (Dₜ): Difference between observed difficulty and student self-reports — captures blind spots
3. Teacher friction (F): Unresolved instructor concerns coded from semi-structured interviews

Topic priority: Pₜ = 0.70·Rₜ + 0.20·Dₜ + 0.10·F

Output is a ranked set of topic priorities with per-topic decision records explaining each ranking.

Key Results

• Instructor alignment: Top-5 mediated topics overlapped with instructor concerns on 3/5 topics; Spearman ρ = 0.80
• Student difficulty alignment: ρ = 0.46 (p = .048) — captures something beyond simple difficulty surveys
• Isolated learner detection: Multi-signal integration AUC = 0.96 vs. 0.91 for gap prevalence alone — identified 2 isolated learners not detected by any single signal
• Construct validity: Reflective thinking, Help-Seeking, and Self-Efficacy aligned with topic understanding scores

Why It Matters

This is one of the first systems to operationalize human-AI co-agency in classroom settings. The interpretable outputs help teachers Trust and act on AI-provided priorities when grades are not yet available. By combining multiple weak signals, the system surfaces students who would otherwise be invisible — a critical capability for equitable instruction.

What this means for practice

• Instructors. Combine several weak process signals instead of acting on one: multi-signal integration separated isolated learners at AUC = 0.96 versus 0.91 for gap prevalence alone, and it surfaced 2 learners that no single signal flagged.
• Instructors. Read the per-topic decision records before acting on the ranking: the top five mediated topics overlapped instructor concerns on 3/5 topics, with rank agreement of ρ = 0.80.
• Instructors. Treat a mismatch between observed difficulty and student self-reports as a prompt to check in with the class, since agreement between mediated priorities and self-reported topic difficulty was modest (ρ = 0.46, p = .048).
• Instructors. Revisit the signal weights with the teaching team rather than treating them as tuned parameters: recommendations held across alternative profiles, with the top-10 topics overlapping 10/10 and removing the disagreement term leaving the top five unchanged (ρ = 0.96).

Limitations

• Single graduate CS course (n=279 surveys, n=5 instructor interviews)
• Preliminary findings — not yet generalizable across diverse contexts
• Weights set by researcher co-design, not learned from data

Citation

Park, J., Medhat, Y., Wai, H. P., Thajchayapong, P., & Goel, A. K. (2026). Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI.