Why Pay for an AI Tutor When You Can Design It? Student-Created AI Agents as Co-Designed Instructional Tools for a Large Software Design and Engineering Course

Synthesis: Zangari, Abramson, Aponte-Archila, Hackney, Muñoz, Omojokun and Feijóo-García (2026) tested a "self-flipped" twist on AI tutoring in a large software design and engineering course (over 700 enrolled): instead of using an AI tutor, students built one. Part A was a scaffolded one-week Dialogflow CX tutorial; Part B was an open-ended one-week assignment to design a conversational agent explaining three design patterns (Strategy, Composite, Observer) to a peer. Of 710 students, 345 voluntarily completed the post-assessment questionnaire. Students valued the open-ended design freedom (22.3% named it the top positive theme for Part B) and learning chatbot skills (17.7%), but the top negative theme was unclear instructions (23.7%), followed by relevance doubts (17.6%) and tool difficulty (12.4%); Part A's tutorial was simultaneously the most-liked (22.0%) and most-disliked (30.0%) element. Correlations between perceived learning and objective quiz scores were real but modest (r = 0.161–0.315 after Bonferroni correction, α = 0.008). The evidence boundary is severe and the authors state it: no control group (sacrificing internal validity by design), volunteer self-selection (n = 345 of N = 710), no quality assessment of the student-created agents, and interrater reliability for the qualitative coding ranging only from fair to moderate (κ = 0.24–0.44). The authors themselves frame the results as "preliminary and exploratory evidence of student engagement rather than causal proof of pedagogical superiority," and flag the alternative explanation that the perceived-learning/objective-score correlations "could potentially reflect students with increased abilities instead of highlighting the strategy's effectiveness."

Summary

As computer science enrollments surge, the widening diversity gap among student populations "underscores a critical need for instructional strategies that are both scalable and equitable." Within large CS courses, strict content pacing and automated grading — frequently implemented as administrative solutions to growing class sizes — "can inadvertently hinder equity by imposing a rigid, one-size-fits-all structure" and scramble the signs of student agency: time investment, self-evaluation, and preparedness. Active learning is the established counterweight: inquiry-based learning, peer instruction, and project-based learning improve conceptual understanding and retention in programming contexts (Amaya Chávez et al. 2020; Córdova-Esparza et al. 2024), with particular benefit for students from underrepresented backgrounds. Student-created learning materials — especially videos — have emerged as a promising way to increase motivation, metacognition, and ownership (Ariza 2023; Vasilchenko et al. 2020), and the "self-flipped classroom" model (Vasilchenko et al. 2018) shows that giving students a more active role strengthens confidence and engagement. This paper extends a prior pilot (Zangari et al. 2025) from two small courses to a large one, testing whether having students co-design AI agents combines the scalability of AI tools with the agency benefits of student-created artifacts.

The design rationale addresses AI tutors' known weaknesses: because many rely on large language models for instant feedback, they "lack personalization," and students hesitate to adopt them "when there are apparent gaps in social norms, usefulness, or even anxiety" (Algerafi et al. 2023; Kuhail et al. 2023). Co-design is proposed as the fix: students prefer agents with clarity, empathy, and responsiveness (Denny et al. 2024), and when preferences are acknowledged in design, tools "are not just competent but conform to student needs." Crucially, the design phase itself is argued to be pedagogically active — students must "bridge the abstract course concepts they're familiar with into teachable pieces," and designing for a peer means "sub-consciously mastering course concepts as they anticipate user needs." The study ran in Spring 2025 at a major Southeastern US university across four sections of an introductory SWE course (over 700 students): Part A (scaffolded Dialogflow CX tutorial, one week) built technical skills but included no course theory; Part B (open-ended, one week) asked each student to individually build an agent explaining the three design patterns to a peer, with encouragement to consider socio-phonetic cues (voice, accent, phrasing, tone). All 57 undergraduate teaching assistants graded with a standardized rubric; grades were not analyzed, to protect anonymity. The post-assessment questionnaire (n = 347 respondents, n = 345 analyzed after excluding 2 self-reported nonsensical responses) combined validated instruments (Heatherton & Polivy performance self-esteem; TIPI personality), Likert perception scales (Diemer et al. 2012), four multiple-choice knowledge questions at Bloom's application/analysis levels, and open-ended likes/dislikes analyzed thematically with consensus coding.

Key Findings

• Engagement themes split sharply between the scaffolded and open-ended parts. Part A positives: Tutorial (22.0%, n = 76 — "clear, step-by-step instructions and scaffolds"), Learning Dialogflow (16.5%, n = 57), Learning Chatbot (13.6%, n = 47). Part A negatives: Tutorial (30.0%, n = 103 — "some students could have struggled with the tutorial or did not find it helpful"), Time (20.1%, n = 69), Irrelevant (13.1%, n = 45 — "It was really long, not relevant to the course, and repetitive"). Part B positives: Open-Ended Nature (22.3%, n = 77 — "the fact that we had relatively few limitations made the assignment kind of fun"), Learning Chatbot (17.7%, n = 61). Part B negatives: Unclear Instructions (23.7%, n = 82), Irrelevant (17.6%, n = 61), Tool Difficulty (12.4%, n = 43). The same tutorial that scaffolded novices was the single most common complaint — structure helps and frustrates simultaneously.
• Perceived learning correlated with objective scores — modestly. After Bonferroni correction for six comparisons per item (α = 0.008), objective quiz score correlated with perceived learning on Part A (r = 0.161, p = 0.003), Part B (r = 0.315, p < 0.001), and perceived performance (r = 0.226, p < 0.001). The authors are careful: "these correlation values, spanning r = 0.161 to r = 0.315, are modest in nature. Their interpretation moving forward is done carefully so as not to exaggerate these findings," and they flag the confound that the correlations "could potentially reflect students with increased abilities instead of highlighting the strategy's effectiveness."
• International students reported significantly higher perceived learning on both parts (Mann-Whitney U, Part A: U(343) = 7515, p < 0.05; Part B: U(343) = 7615, p < 0.05) — the only demographic showing significant perception differences; most other demographics showed no significant differences (p > 0.05).
• English learners spent significantly more time on Part A (the scaffolded tutorial) than native speakers (Mann-Whitney U, U(343) = 11838.5, p < 0.05), with no significant difference on Part B — suggesting scaffolding load, not the creative task, carried the language burden. Academic year also mattered: first-year students differed significantly from third-year students on Part B dedication (Kruskal-Wallis H(3) = 9.87, p < 0.05; Dunn's pairwise p < 0.05), with third-years reporting higher time.
• Personality and self-esteem patterns carry an equity warning. Performance self-esteem correlated positively with extroversion (r = 0.262, p < 0.001) and with objective scores (r = 0.232, p < 0.001), but not with extroversion-and-perceptions jointly (r = −0.05, p = 0.356). Gender differences in self-esteem were significant (Kruskal-Wallis H(2) = 9.87, p < .005): female students reported significantly lower performance self-esteem than male peers (Dunn's p < .005). The authors' design inference: "instructors in large software engineering courses must be mindful of introverted students or those from underrepresented backgrounds, ensuring that open-ended design tasks include sufficient scaffolding to equitably build performance self-esteem across all personality types."
• The most common positive theme was creative freedom; the most common negative was ambiguity. Open-endedness (22.3%) topped Part B positives while unclear instructions (23.7%) topped its negatives — the design lesson the authors draw: "balance the novelty of these activities with clear connections to course objectives and professional skills, and provide enough support to address potential challenges."
• Interrater reliability was fair-to-moderate, and the authors say findings should be read accordingly. Independent-rater agreement on 73 responses ranged from κ = 0.44 (dislike questions, both parts — moderate) to κ = 0.24 (Part A likes — fair): "some categories required subjective interpretation... findings derived from these qualitative codes should be interpreted with appropriate caution."

What this means for practice

• Building the tutor can be the pedagogy. The core transferable idea: making students design an AI agent for a peer forces them to "bridge the abstract course concepts... into teachable pieces," anticipate user needs, and practice communication — the design work itself is argued to produce the metacognitive and ownership benefits that student-created videos show (Vasilchenko et al. 2018, 2020). For instructors, this reframes the question from "should students use an AI tutor" to "what does having students build one teach."
• Pair the open-ended assignment with the scaffolded on-ramp, and connect both to the course. The Part A/Part B split worked because Part A built the tool skills Part B required — but 13.1% and 17.6% of students found the parts irrelevant to the course, and Part B's top complaint was unclear instructions. The authors' recommendation is explicit: "balance the novelty of these activities with clear connections to course objectives and professional skills."
• Scaffold open-ended tasks for the students least likely to feel confident in them. The extroversion–self-esteem correlation and the gender self-esteem gap are the actionable findings: open-ended, conversational, creative tasks advantage students who already feel confident. Sufficient scaffolding is framed as an equity requirement, not a pedagogical nicety.
• Budget for tool-learning time, and watch who pays it. English learners spent significantly longer on the scaffolded tutorial; Part A's time complaint (20.1%) was the second-most-common negative. Tool choice (Dialogflow CX was picked partly for its gentle learning curve) and explicit time expectations matter for adoption.
• Read the evidence at its actual strength. This is a no-control-group, volunteer-sample, exploratory study of perceptions — valuable as "a valuable sample to examine the benefits and challenges" and as motivation for further research, not as evidence that the strategy outperforms alternatives. Any instructor adopting it should treat the effect claims (perceived learning ↔ objective scores) as hypotheses for their own measurement.

Limitations

• No control or comparison condition. The authors acknowledge this "sacrifice[s] the study's internal validity" — chosen so the strategy fit the course's natural pedagogical structure, but it means no causal claim about the strategy versus any alternative is possible.
• Volunteer self-selection. Only students who "voluntarily decided to participate and successfully completed the study tasks (n = 345)" of N = 710 are represented; the authors call for future work to compare participant demographics and prior performance against the full course population "to accurately quantify and mitigate potential self-selection bias."
• Student-created artifacts were never assessed for quality, accuracy, or effectiveness. The authors acknowledge this gap — "no anomalous submissions or conditions were observed" is not a quality measure — and plan future work "to fully connect artifact outcomes with student experiences."
• The objective-score instrument is exploratory. The authors flag that the four multiple-choice questions are exploratory and that "future studies should focus on using more structured instruments already trusted by the community."
• Fair-to-moderate interrater reliability on qualitative coding (κ = 0.24–0.44), particularly on the "likes" questions, limits confidence in the specific theme frequencies.
• Single course, single institution, single tool, single term. One large introductory SWE course at one Southeastern US university using Dialogflow CX in Spring 2025; the authors note future iterations "may consider navigating other course formats (e.g., large or small) and exploring the inclusion of strategy like ours in previous and more senior CS courses."

Citation

Zangari, L. G., Abramson, S. L., Aponte-Archila, E., Hackney, K. A., Muñoz, E., Omojokun, O., & Feijóo-García, P. G. (2026). Why Pay for an AI Tutor When You Can Design It? Student-Created AI Agents as Co-Designed Instructional Tools for a Large Software Design and Engineering Course. ASEE Computers in Education, 15(4), 311–329.