🧠 AI Ed Wiki

📄 Research Article

Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents

● high · 2026-08-03llm, collaborative-learning, learning-analytics, student-ai-interaction, nlp-education, collaborative-ai-tutoring, metacognition, ai-tutoring

Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas — CogSci 2026 (accepted full paper).

Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas — CogSci 2026 (accepted full paper).

Synthesis

This study applies an extended 7-point ICAP framework (Interactive, Constructive, Active, Passive) to characterize cognitive engagement in collaborative dialogue, comparing trained human annotators with LLM-based labeling: in-context learning (ICL), zero-shot prompting, and self-reflective agents.

Human interrater reliability was robust across framework refinement stages (kappa = 0.906–0.998), far exceeding ICL-based annotation (kappa = 0.541–0.609) — a large gap between human and LLM labeling of engagement.

The human-refined framework improved human agreement (Δkappa = 0.10) but gave only modest gains to ICL LLMs (Δkappa < 0.04); agent-refined frameworks improved cross-model agreement but stayed below human-refined performance.

Findings highlight the promise of reflective-agent approaches for scaling engagement measurement while underscoring that LLM annotation of learning processes still trails trained humans — relevant for learning analytics pipelines that rely on automated discourse coding.

Connected Concepts

  • Adaptive Learning
  • Human In The Loop AI
  • Knowledge Tracing
  • Administrator
  • Help Seeking
  • Pedagogical Agent
  • Automated Question Generation
  • RAG

    • Connected Articles

  • AI Agents Peer Learning Discourse — When AI Agents Teach Each Other: Discourse Patterns Resembling Peer Learning in the Moltbook Community
  • LLM Facilitation Timing Online Discussions — To Facilitate or not to Facilitate: Human and LLM Facilitator Tendencies in Online Discussions
  • AI Peer Feedback Systems — AI Peer Feedback Systems
  • Hingle Collaborative AI Literacy 2025 — Collaborative AI Literacy Framework
  • Spritz AI Disciplinary Mediation Student Teams 2026 — Exploring AI-Supported Disciplinary Mediation in Student Project Teams' Text-Based Communication
  • Prober AI Inquiry Writing — Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing

    • Citation

    Do, L. A., Jiang, H., Aeron, S., & Thomas, A. K. (2026). Measuring cognitive engagement in collaborative discourse with an extended ICAP framework. CogSci 2026. arXiv:2607.28651.