You've Got AI Friend in Me: LLMs as Collaborative Learning Partners

Synthesis: You've Got AI Friend in Me: LLMs as Collaborative Learning Partners — This exploratory classroom study (n = 154) tested whether Large Language Models can serve as collaborative learning partners. Across one semester, introductory social science students wrote argumentative essays, had them critiqued by LLMs such as ChatGPT, Gemini, or Claude, and either incorporated or rebutted the critiques. Students engaged deeply with the LLMs, enjoyed the work, and showed gains in argumentative writing, prompt engineering, response-to-feedback quality, and Self-Efficacy for working with generative AI.

Key Findings

• Students improved on all three performance dimensions from the first to the fifth iteration of the assignment: quality of initial argument (2.58 → 2.90, t(109) = 6.22), prompt engineering (2.34 → 2.89, t(109) = 8.80), and response to AI Feedback (2.41 → 2.88, t(109) = 5.44); all p < .001, with improvement roughly a full standard deviation per dimension.
• Gains appeared even in essays written without LLM support. Students' initial-draft argument quality improved over the semester, suggesting skill development rather than mere tool dependency on the LLM — though causal attribution is limited by the lack of a control condition.
• Students engaged deeply rather than passively. Blind coders found evidence of reflection in 92.7%, rebuttal of LLM claims in 87.8%, and acceptance of feedback in 93.6% of responses (inter-rater κs = 0.81–0.89), indicating students acted as critical consumers who actively decided what to accept or refute.
• Self-efficacy for generative AI grew across the semester. Current self-efficacy was significantly higher after the fifth iteration than the first (t(177) = 4.25, p = .006), with the strongest within-assignment growth on the final iteration (t(97) = −5.65, p < .001).
• Affective responses were positive and durable. Students rated the assignment useful, engaging, enjoyable, and appropriately challenging at both the first and fifth iterations (all significantly above the neutral midpoint, ps < .002), with engagement and enjoyment stable across the semester.
• A qualitative sentiment analysis (run via GPT-4) classified 92.9% of general open-ended responses as positively valenced, reinforcing the quantitative affective results.

What this means for practice

• Instructors. Grade the response to the AI, not just the revision. The assignment awarded points for how effectively students rebutted the model's critique or updated their argument from it, and blind coders found rebuttal in 87.8% and acceptance in 93.6% of responses — evidence that the prompt design, not the tool, produced the critical engagement.
• Instructors. Give students explicit prompt-engineering guidance at the start of the semester and assess against it. Prompt engineering was one of three graded dimensions and improved across iterations alongside argument quality and response to feedback.
• Instructors. Keep one iteration unsupported by an LLM so that unaided writing can be observed; students' initial-draft argument quality improved across the semester in essays written without model support.
• Designers. Scaffold the workflow in fixed stages (independent draft, model critique, fact-check of the model's claims, then revision) and let students drop their lowest iteration — the course allowed students to skip one of five graded cycles without penalty.
• Designers. Budget for a writing-intensive rhythm across a whole semester rather than a single assignment: gains in this study accumulated over five iterations of the same scaffolded task.

Limitations

• There was no control condition. Every student in the single introductory social science elective (n = 154) worked with an LLM, and the authors state that other course elements — lectures, discussion, or generative-AI practice outside class — may account for part of the growth, so causal attribution is not possible.
• All students sat in one class graded on a single curve, which is also why differential treatment was ruled out on logistical, pedagogical, and ethical grounds; the study is therefore an exploratory classroom design rather than a randomized trial.
• The performance measures come from one teaching assistant's rubric scoring of two assignments (iterations 1 and 5), and although the coder was blind to the study he was not blind to assignment timing, so fatigue or leniency as the term progressed could inflate the apparent gain; the first task could also have been easier than the last.
• The Self-Efficacy and affective findings are self-report, using bi-valenced Likert scales and open-ended text, and the sentiment classification used to corroborate them was itself run with GPT-4 — the same class of model under study.

Citation

Oppenheimer, D. M., Cash, T. N., & Connell Pensky, A. E. (2025). You've Got AI Friend in Me: LLMs as Collaborative Learning Partners. International Journal of Artificial Intelligence in Education, 35, 3896–3921.