Is Solving Better Than Evaluating GenAI Solutions?

Randomized A/B crossover study (N=220) in a junior-level algorithms course comparing solution evaluation/critique tasks against traditional solution generation. Finds that evaluation-centered tasks produce comparable learning outcomes with a localized homework advantage that does not transfer to summative performance, suggesting evaluation tasks are a viable pedagogical response to the availability of GenAI code generation tools.

Key Findings

No statistically significant differences were found between groups on midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions.

Students received significantly higher homework scores during periods in which they evaluated GenAI-generated solutions, but this localized advantage on the modified homework items did not translate into downstream summative gains.

Survey data indicated that most students did not report changing their study habits in response to the intervention; however, students who did report adapting their study strategies rated the GenAI-evaluation assignments as significantly more helpful.

GenAI evaluation redistributes student effort from open-ended solution construction toward verification, diagnosis, and judgment, but does not automatically produce stronger conceptual transfer.

GenAI-evaluation activities can be incorporated into algorithms coursework without broad performance losses, but meaningful learning gains may require deliberate scaffolding that pushes students beyond simple error diagnosis.

Study Design & Method

The study was a randomized A/B crossover experiment (N = 220) conducted in a junior-level, theory-heavy algorithms course. Across six assignments, student working groups either solved challenging algorithmic problems directly or evaluated often-flawed GenAI-generated solutions to corresponding problems, with group roles reversed midway through the semester. A survey with a high response rate (Group A: n = 101; Group B: n = 107; overall N = 208, 95%) measured students' self-reported comfort and confidence across the five core algorithmic topics assessed on the midterm — asymptotic analysis, divide and conquer, dynamic programming, greedy methods, and graph algorithms — using five-point Likert scales. Within-student difference scores were computed to examine changes in homework performance on GenAI-graded versus independently solved problems across the two halves of the course.

Implications for AI in Education

The results speak directly to the debate over how computing educators should respond to GenAI tools that can generate assignment solutions. Evaluation-centered tasks — verification, critique, and diagnosis of AI-produced artifacts — are a defensible pedagogical option: they can be incorporated without broad performance losses, and students who engaged with them as a strategy reported greater perceived value. However, the absence of transfer to exams warns that evaluation practice alone does not build the same conceptual depth as generation under struggle, and the authors conclude that meaningful gains require deliberate scaffolding beyond simple error diagnosis. For CS Education and Critical Thinking instruction, the study suggests framing GenAI evaluation as a structured activity with explicit reflection, rather than assuming that judging flawed solutions automatically strengthens algorithmic understanding. It also models a rigorous A/B crossover methodology for evaluating GenAI pedagogy in Higher Ed.

Connected Concepts

Connected Articles

Cognitive Shift AI Education — Evidence of a Cognitive Shift in AI Education: How Students Are Rethinking Human Intelligence?

Learner AI Interaction Patterns Oop — Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course

AI Changing Teaching Workflows — How AI Is Changing Teaching Workflows

Trio Ethnography LLM Programming Education — Beyond Perspectives: A Trio-Ethnography of Interpretation Evolution in LLM-Supported Programming Education

AI Generated Instructional Videos Computing Ed — Student Perceptions and Preferences Regarding AI-Generated Instructional Videos in Computing Education

GenAI Availability Grades Satisfaction — Generative AI Availability, Grades, and Student Satisfaction at a Large University

Citation

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas (2026). Is Solving Better Than Evaluating GenAI Solutions?. arXiv:2607.27586. cs.CY, cs.AI.