A Feasibility and Implementation Integrity Study of the Community Builder (CoBi): An AI-based Collaboration Support System in K-12 Classrooms

Synthesis: A Feasibility and Implementation Integrity Study of the Community Builder (CoBi): An AI-based Collaboration Support System in K-12 Classrooms — This paper introduces the Community Builder (CoBi), a classroom-wide AI system that uses speech recognition and language understanding to create non-evaluative, privacy-preserving visualizations of uplifting small-group student discourse, intended to support students' Collaborative Learning skills and community building. Through an interview study with three middle school teachers and 12 students plus two classroom pilots with 61 students across six K-12 classrooms, the authors establish broad student and teacher buy-in, technical feasibility in noisy real-world classrooms, and the central finding that high-integrity use depends on substantial Workplace Learning for teachers. The work surfaces transferable design principles around Teaching facilitation, Human-in-the-Loop orchestration, and navigating tensions between privacy, surveillance, and student agency.

Key Findings

• CoBi detects expressions of "uplifting" discourse (e.g., "that's a great idea!") mapped to Community Agreements (CAs) across four categories—being respectful, being equitable, being committed to community, and moving our thinking forward—and displays aggregated, classroom-level feedback as non-evaluative Visualizations (an organic tree animation, a radar chart, and a sentence cloud), deliberately withholding student- or group-level feedback to protect privacy and build Trust.
• In the interview study (Study 1), both students and teachers expressed broad buy-in for an AI system that could support the relational dimension of collaboration; 83.3% of student pairs preferred the qualitative tree over the quantitative radar, students emphasized Privacy concerns (including a desire that CoBi not "snitch" to administrators), and teachers worried about classroom integration and being put "on the spot" by real-time feedback, favoring pre/post-action review over live display.
• The technical feasibility study (Study 2) with CoBi v1 across three classrooms showed the Edtech Platform could be deployed successfully in noisy middle school environments using commodity microphones and a scalable cloud pipeline; however, time constraints and classroom management cut the revisiting phase short, and mismatches between teacher and student interfaces created confusion about whether visualizations were group- or class-level.
• The implementation integrity study (Study 3) with CoBi v2 found teachers pursued diverse adaptations with varying fidelity to design goals: high-integrity engagements used noticings and Visualizations to spark Metacognition and critical reflection about collaboration and the AI's classifications, while low-integrity uses drifted toward using CoBi as a performance-monitoring device or veering into general AI Literacy discussions.
• CoBi's discourse classification relies on three fine-tuned RoBERTa transformer models trained on human- and Whisper ASR transcripts, achieving AUROCs of 0.84, 0.77, and 0.82 on human transcripts (0.71, 0.67, 0.71 on Whisper transcripts), adequate for Educational NLP-based class-level feedback via the principle of aggregation.
• The authors identify four transferable design principles: (1) significant Teaching professional learning is required for integral use; (2) real-time multi-device feedback must be managed to avoid fragmenting student attention; (3) classroom-level (rather than individual) feedback effectively navigates tensions between Privacy, surveillance, and student agency; and (4) positioning ASR imperfections as a feature enables productive discussion of AI's affordances and limitations, supporting Trust Calibration.

What this means for practice

• Instructors. Plan substantial professional learning and classroom routines around CoBi's three phases rather than expecting the display to speak for itself; high-integrity use — turning noticings and visualizations into Metacognition and critique of the AI's own classifications — depended on teacher learning, while teachers not given CoBi-specific guidance drifted toward performance monitoring.
• Instructors. Schedule the revisiting discussion as a pre-action review at the start of the next class instead of at the end of the current one; in the v1 pilot, time constraints and classroom management cut the end-of-class revisiting phase short.
• Instructors. Use the system's errors deliberately. Teachers who treated misclassified noticings and ASR garble as material for discussing what the AI can and cannot do converted a technical weakness into AI Literacy instruction.
• Designers. Keep feedback at the class level and framed as non-evaluative. Aggregated tree, radar, and sentence-cloud visualizations were what made the privacy tension manageable — students explicitly objected to anything that would "snitch" to administrators — and 83.3% of student pairs preferred the qualitative tree over the quantitative radar.
• Designers. Align teacher and student interfaces before deployment. In the pilot, mismatched views left teachers and students unsure whether a visualization described a group or the whole class.

Limitations

• No efficacy evidence by design: the authors state the goal was buy-in, technical feasibility, and implementation integrity rather than efficacy, and that randomized controlled trials remain future work — there is no control condition and no measure of whether students' collaboration skills improved.
• Consent sharply bounded the samples: study 2 analyzed 24 fully consented students and study 3 analyzed 37 of 97 enrolled students, three classrooms each in one school district, while study 1 drew on 3 STEM/Science middle school teachers and 12 seventh graders recruited opportunistically from two schools.
• The discourse classifiers were trained on data from a single curriculum unit, cover only three of the four Community Agreement categories, and lose accuracy on Whisper ASR transcripts (AUROCs of 0.71, 0.67, and 0.71) relative to human transcripts (0.84, 0.77, and 0.82).
• CoBi processes speech only: the authors state it cannot attend to non-verbal cues such as gaze and head movement that carry conversational feedback, with multimodal support slated for a future version.

Citation

Breideband, T., Bush, J. B., Reitman, J. G., Rose, S., Weatherley, J., Penuel, W. R., & D'Mello, S. K. (2025). A Feasibility and Implementation Integrity Study of the Community Builder (CoBi): An AI-based Collaboration Support System in K-12 Classrooms. International Journal of Artificial Intelligence in Education, 35(4), 3579-3613.