🧠 AI Ed Wiki

📄 Research Article

MindCopilot: Towards Formalizing and Evaluating Granular Human-LLM Co-Writing

● high · 2026-05-25feedback-loop, higher-ed, human-in-the-loop, llm, writing-education, human-in-the-loop-ai

MindCopilot introduces a formal framework for evaluating human-LLM co-writing that shifts from output-only metrics (BLEU, ROUGE) to interaction-aware evaluation. The paper models co-writing as a Human-in-the-Loop Markov Decision Process (HiL-MDP), where writing is a sequence of granular decisions: accept, edit, or reject each AI suggestion. The Co-Writing Fidelity Suite introduces two novel metrics — Hierarchical Acceptance Rate (quantifying acceptance across granularity levels) and Knowledge-aware Editing Distance (measuring cognitive effort of editing suggestions). Evaluated across 1,688 queries from 16 writing domains, with a 30-participant validation study.

For Writing Education, this represents a paradigm shift. Traditional automated writing evaluation focuses on final text quality, missing the process-level decisions that constitute real learning. The HiL-MDP framework aligns with process-oriented writing pedagogy — capturing where students accept, modify, or reject AI suggestions provides rich diagnostic information beyond what a final essay grade reveals. The work connects to Cyberscholar GenAI Writing Feedback and Structured LLM Feedback Programming by establishing that interaction structure (when/how suggestions appear) significantly affects user behavior. For AI Peer Feedback Systems, the acceptance/editing metrics offer a template for evaluating collaborative feedback workflows. Accepted to IJCAI 2026, this work bridges Human In The Loop AI theory with practical writing tool design, advancing LLM-based educational technology beyond static quality assessment toward dynamic interaction modeling.

Connected Concepts

  • Writing Education
  • Human In The Loop AI
  • LLM

    • Connected Articles

  • Cyberscholar GenAI Writing Feedback
  • Structured LLM Feedback Programming
  • AI Peer Feedback Systems

    • Citation

    Youqing Fang, Yinhao Tang, Yanan Sun, Jiangning Liu, Ziyi Wang, Xun Zhao, Bin Liu, Weiming Zhang, Kuikun Liu, Wenwei Zhang, Kai Chen (2026). MindCopilot: Towards Formalizing and Evaluating Granular Human-LLM Co-Writing. arXiv:2605.23535. IJCAI 2026. - Humanlike AI Collaborative Writing — Anthropomorphic design choices amplify both benefits and costs of AI co-writing