Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming

Synthesis: Chen et al. (2026) ran a 2×2 between-subjects experiment with 24 Canadian undergraduates to test whether access to ChatGPT helps students learn a new programming concept or merely finish the task. Beginners with ChatGPT passed far more of 500 generated tests than beginners without it (410.7 vs 80.7, p=0.015), while intermediate performance barely moved (291.0 vs 290.8). Understanding ran the other way: four of six beginners with ChatGPT gave weak explanations of their own solutions, and intermediates without the tool gave the strongest explanations. The authors read this as students using generative AI as a problem-solving rather than a learning tool, and argue that both heavy reliance and minimal use produced weaker knowledge gains in computing education.

Key Findings

1. Complete solutions won the task, not the concept. Among the twelve participants who had ChatGPT, those who generated whole solutions averaged 495 of 500 tests with five of six passing every test, against 206.7 and one of six for stepwise generation (p=0.032). Only one of those twelve, P3, showed strong understanding.
2. Beginners gained most in score. Beginners with ChatGPT averaged 410.7 passed tests against 80.7 without it (p=0.015). Intermediates were effectively unchanged by access to the tool, at 291.0 with ChatGPT and 290.8 without.
3. Beginners' explanations ran the other way. Four of six beginners with ChatGPT were rated weak on understanding their own solution, against one of six beginners without it. The no-ChatGPT beginners explained their reasoning more clearly even when their code failed to compile, as it did for P14 and P15.
4. Intermediates without the tool understood most. Four of six intermediates without ChatGPT reached strong understanding of their solution and three of six reached strong understanding of the concept. With ChatGPT, half were partial and half weak on both measures.
5. Concept understanding was the one place beginners gained. Beginners with ChatGPT split one strong, two partial, and three weak on the concept, against five partial and one weak without it. The authors connect this to four beginners asking for concept explanations, compared with one intermediate.
6. Use was narrow and both extremes failed. All six beginners and four intermediates used ChatGPT for API or syntax questions, while bug detection appeared once per group and test generation once overall; only 33% asked for code or conceptual explanations. P4 refused a full solution without understanding the concept, P9 used the tool only for syntax, and P1, P2, P6, and P10 outsourced the task and explained little.

What the study did

The experiment was run at a Canadian university with 24 undergraduates recruited by email and posters in introductory and intermediate programming courses, each paid $30 CAD regardless of performance. Participants were split into beginner and intermediate levels and then into ChatGPT and no-ChatGPT conditions, six per cell. Prior programming knowledge separated the two groups: beginners had completed only introductory programming and data structure courses, while intermediates were in third- or fourth-year software engineering or computer science programs or reported 3+ years of coding.

Everyone watched a 10-minute video on the dynamic sliding window technique, a concept the authors verified was outside the standard curriculum and unknown to all participants, then had 30 minutes to solve a LeetCode-style problem with test cases available. The ChatGPT condition could use only GPT-3.5-turbo as an external resource; the comparison condition could use documentation and sites such as Stack Overflow but no AI. Performance was measured with 500 test cases generated by Randoop from a verified reference solution.

Knowledge gains were not measured by a test. Two open-ended post-survey questions asked participants to explain their approach and to relate their solution to the video concept, and two authors coded the answers as strong, partial, or weak understanding, with Cohen's kappas of 0.81 and 0.84. Chat histories and screen recordings supplied the interaction analysis. The paper is an arXiv preprint, revised in June 2026.

Why the two groups diverged

The interaction records show different intents rather than different tools. Four of six beginners generated complete solutions, against two of six intermediates, the behavior the authors treat as over-reliance; intermediates were more likely to build code stepwise and to keep their own plan in the lead. Concept explanation went the other way, four beginners against one intermediate, which the authors offer as a partial explanation for why beginners with ChatGPT were the only group to improve on concept understanding.

The authors explain the split by timing of exposure. ChatGPT became widely available in late 2022, while most intermediate participants began programming in 2021 or earlier. Beginners could therefore meet GenAI before they had built programming foundations or learned to work from documentation and forums, and they treated it as a primary problem-solving tool. Intermediates had already learned through conventional resources and folded the tool in more cautiously. That account is plausible but post hoc: proficiency level and exposure history are confounded in a design with six participants per cell.

The transcripts also show that strategy, not access, separated the outcomes. P1 and P2 asked ChatGPT to explain the concept and still could not connect the generated solution to it. P3 used the same complete-solution strategy but read the explanations closely and could articulate both the code and the concept. P4, P8, and P9 worked stepwise but built on flawed plans, so a more interactive process did not rescue weak reasoning.

What this means for practice

• Instructors: grade the explanation alongside the artifact. Working code and weak explanation coexisted for four of six beginners with ChatGPT, so a passing test suite is not evidence that the concept landed.
• Instructors: teach a strategy rather than a policy. The single strong case among the twelve, P3, used complete-solution generation but read and interrogated the explanation; the authors argue against an all-or-nothing stance of banning the tool or allowing unrestricted, unguided use.
• Designers: make conceptual explanation the default request. Only 33% of participants asked for code or concept explanations without prompting, and concept explanation was requested by four beginners against one intermediate.
• Administrators: do not read rising pass rates as learning. Beginner scores rose from 80.7 to 410.7 passed tests while beginners' solution explanations became weaker.
• Researchers: test understanding inferentially and after a delay. The paper reports p-values only for task performance and codes understanding descriptively, so the knowledge-gain comparison rests on distributions of six participants per cell.

Limitations

The study has 24 participants in four cells of six, at one Canadian university, on one dynamic sliding window task in a single 30-minute session with one model, GPT-3.5-turbo. Knowledge gains come from two open-ended survey questions coded into three levels, not from a test of the concept, and the paper reports no inferential statistic, confidence interval, or effect size for them; only the performance comparisons carry p-values. There is no delayed post-test, so nothing here shows whether understanding persisted or transferred, and the coding of explanations, while reliable between raters, measures what students could say immediately after finishing. The timing-of-exposure explanation for the beginner and intermediate split is the authors' interpretation of a between-subjects difference, not a manipulated variable, and prior experience with ChatGPT was not controlled.

Citation

Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2026). Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming. arXiv:2511.13271v2 [cs.SE]. Preprint.