AIEd Knowledge Base logoAIEd Knowledge BaseAIEd KB
On this page

Research Article

Co-designing AI with youth partners: Enabling ideal classroom relationships through a novel AI relational privacy ethical framework

Synthesis: Chang et al. (2025) argue that designers carry an ethical responsibility to engage youth — especially historically minoritized K 12 students — as design partners in AI tools from conception, and demonstrate this through the Learning Futures Workshop, a participatory study that brought 30 historically minoritized youth into conversation with education and technology experts. The workshop surfaced students' hopes for expansive, collaborative possibilities with AI in classrooms, alongside a central tension around the data collected when AI agents mediate collaboration. Analyzing workshop findings through Nissenbaum's contextual integrity framework, the authors identify three ethical commitments and derive a novel AI Relational Privacy ethical framework to guide the equitable design of AI-supported collaboration tools. The framework is operationalized in Community Builder (CoBi), a tool supporting students in building customized, ideal collaborative relationships. The paper argues that tools designed without students as partners risk being untrustworthy and inequitable.

Co-Designing AI With Youth as Partners

Much of the design of AI-based educational tools has been driven by researchers projecting their own expertise and perspectives, leaving youth agency limited. This paper contends that designers have a significant ethical responsibility to bring students' dreams and concerns into the design of AI tools starting at conception — a need made urgent as applications like AI-supported collaboration introduce new surveillance vectors into K 12 classroom spaces. Rather than designing at students, the authors position youth as co-designers who imagine ideal classroom relationships that AI could support.

The work draws on youth participatory design (Druin), human-centered design, and relational approaches from the learning sciences. Key to their framing is re-mediation rather than remediation: shifting youth from a "delegitimized stakeholder" position to a meaningful design partner, and re-imagining the relationships that exist in the institutions being designed for. The workshop, run by the Institute for Student-AI Teaming (iSAT), explicitly communicated the tension between youth's expansive hopes and existing technical expertise.

The Learning Futures Workshop

The Learning Futures Workshop was held remotely over five days during summer 2021 with 30 high-school youth (grades 9–12) from California, Colorado, and Oklahoma, including students identifying as Asian-American/Pacific Islander, Latinx, African American, Native American, and white. Over the first two days youth learned about the affordances and limitations of supervised machine learning and applied a sociopolitical lens to AI. The final three days were spaces for dreaming, designing, and enacting AI possibilities.

  • Day 3 (Dreaming): Youth imagined ideal collaboration and how AI might realize it. They surfaced hopes for friendship finders, collaboration matchers based on shared interests, AI that keeps disruptive peers accountable, and an emphasis on "equity not equality" in collaboration.
  • Day 4 (Data): Using a custom privacy worksheet, youth considered the embodiment, actions, inferences, and raw data required by their proposed agents — and explicitly what they hoped the AI would not do. Discussions revealed discomfort with AI collecting personal data beyond school boundaries and preferences that varied by context and over time.
  • Day 5 (Enacting): A modified Theatro activity grounded AI in familiar collaborative scenarios. Youth discovered that some AI possibilities require sharing data with authoritative others (teachers, administrators, parents) to work effectively, even when that violates their privacy preferences.

Contextual Integrity and Relational Privacy

The authors analyze workshop findings through Nissenbaum's framework of contextual integrity, which holds that information flows within social contexts parameterized by sender, recipient, information types, and transmission principles. They model AI-supported collaboration with these tuples and draw on Gutiérrez's notions of first, second, and third spaces in classrooms to illuminate how collaborative contexts carry different privacy expectations.

Three key ethical commitments emerged from the findings: designing for ideal classroom relationships; recognizing that youth privacy preferences may conflict with their ideal designed relationships; and acknowledging that some AI tools require support from classroom authority figures to function effectively — support that can compromise relational privacy.

From these, the authors derive a five-question AI Relational Privacy ethical framework to guide design: What ideal relationships does the AI support? What do designers/secondary recipients consider key information types and transmission principles? What do youth (senders) consider preferred information types and transmission principles? Does the design require data sharing beyond the immediate context, and does that violate initial transmission principles? How are norms contested and re-negotiated, and how does that expand or narrow the design?

Community Builder (CoBi)

The framework is instantiated in the Community Builder (CoBi), an AI tool developed at iSAT that supports students in building customized, ideal collaborative relationships. CoBi uses computational models to give feedback on how well students hold to collaborative agreements such as Committed to Community, Moving Thinking Forward, and Being Respectful — agreements rooted in the relational hopes expressed by youth in the workshop. Applying the Relational Privacy framework surfaced key tensions: for example, an AI that reports off-task remarks to teachers supports "Moving Thinking Forward" but compromises "Being Respectful." CoBi has addressed this tension by aggregating predictions at the class level to protect individual and group identities, illustrating the need for compromise rather than consensus.

Implications

The paper argues that students who will live with these tools must be engaged as design partners, or tools risk being untrustworthy, inequitable, disposable, and ineffective. Future work focuses on changing the organizational conditions through research-practice partnerships, professional development that embraces the classroom's "third space," and curriculum that helps youth understand how tools like CoBi collect, process, and secure data. This stands as the first study to explore the privacy implications of AI-supported collaboration in K-12 contexts in close design partnership with historically minoritized youth.

Connected Concepts

Connected Articles

Citation

Chang, M. A., Tissenbaum, M., Philip, T. M., & D'Mello, S. K. (2025). Co-designing AI with youth partners: Enabling ideal classroom relationships through a novel AI relational privacy ethical framework. Computers and Education: Artificial Intelligence, 8, 100364.

View SourceView ChangesEditChat