The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

Synthesis. Niousha et al. (2026) argue that AI tutor evaluation, which conventionally judges only the pedagogical quality of feedback, is missing a critical axis: what students actually do with that feedback. They propose an engagement-based evaluation framework — grounded in observable revision behavior — that measures whether students act on tutor feedback and whether those actions are applied correctly. Applied to 10,235 real code submissions across two semesters of an introductory programming course at UC Berkeley, the framework reveals substantial differences between two deployed AI tutors that pedagogy-only evaluation could not distinguish. Crucially, these behavioral signals (feedback relevance and success) are more strongly associated with students' perceived helpfulness of feedback than pedagogical quality alone, offering a more complete and actionable picture of AI tutor performance.

Definition

A framework for evaluating AI tutoring systems that extends beyond the pedagogical quality of feedback to measure what students actually do with that feedback — whether they act on it and whether they apply it correctly. Proposed by Niousha et al. (2026) based on analysis of 10,235 real student code submissions.

Key Findings

1. Pedagogy-only evaluation is insufficient: Two AI tutors with similar pedagogical quality showed dramatically different student engagement patterns that pedagogy-only metrics could not capture.
2. Behavioral signals predict perceived helpfulness better: Engagement-based metrics (feedback relevance and success) are more strongly associated with students' perceived helpfulness of feedback than pedagogical quality scores alone.
3. Engagement adds separability between tutors: The misconception-focused tutor achieved 9–21 percentage-point higher feedback relevance on every assignment, a difference invisible in pedagogical scoring.
4. Actionable metrics: The framework provides concrete measurements — action rate (did the student modify their submission?) and correct application rate (was the modification applied correctly?) — that jointly reveal whether feedback drives learning.

Significance for AI in Education

This work addresses a critical evaluation gap in AI Ed Evaluation. An AI tutor that gives perfect pedagogical feedback is worthless if students ignore it or apply it incorrectly. The behavioral axis complements pedagogical assessment to provide a complete picture of real-world effectiveness, and challenges assumptions in The Evidence Base on AI in K-12: A 2026 Review about what makes tutoring effective.

How the Evaluation Works

The study situates evaluation in the student–AI tutor interaction within CS61A, an introductory programming course at UC Berkeley with roughly 1,000 students per semester. Students solve problems in an online environment with an autograder that gives immediate feedback on passed and failed test cases; whenever a submission fails, an LLM-based tutor returns natural-language feedback. The dataset spans Fall 2024 (BaselineTutor) and Fall 2025 (MisconceptionTutor), two tutor configurations built on the same LLM (GPT-4) that differ only in prompting structure — the latter adding explicit detection and targeting of likely student misconceptions from an instructor-authored list.

Two Complementary Metric Families

• Pedagogy-based evaluation scores feedback along eight learning-science-motivated dimensions — mistake identification, mistake location, answer revealing, guidance, actionability, coherence, tone, and humanness — summarized as a Desired Annotation Match Rate (DAMR) computed by an LLM judge.
• Engagement-based evaluation quantifies whether feedback was used (relevance) and whether it was applied correctly (success). An LLM judge attributes each feedback sentence to the student's subsequent code edit, yielding RelScore (fraction of feedback engaged) and SuccScore (fraction of engaged feedback applied correctly), with substantial human agreement (κ ≈ 0.80–0.89).

Results and Interpretation

Across all pedagogical dimensions and assignments, MisconceptionTutor achieved higher DAMR than BaselineTutor, most differences statistically significant. It adopted a more conservative strategy, reducing answer revealing (99.65 vs. 93.95 DAMR) at the cost of a small drop in immediate actionability — a trade-off suggesting that high-level, misconception-focused feedback can sacrifice quick next steps.

The engagement analysis tells a different story. MisconceptionTutor improved feedback relevance by 9–21 percentage points on every assignment — a consistently larger fraction of its feedback influenced students' subsequent code edits, a gap never visible in pedagogical scoring. Success improvements were concentrated on earlier assignments and diminished or reversed on later, more difficult ones, indicating that engaging feedback does not always translate into immediate correct application on challenging problems.

A striking finding concerns answer revelation: on both tutors, feedback marked as undesired on the revealing-answer dimension achieved higher success scores than desired feedback (e.g., 79.4% vs. 53.0% for MisconceptionTutor). This suggests such successes were driven by students copying a revealed answer rather than understanding and solving the problem themselves — a caution that high success scores alone do not guarantee learning.

In binary logistic regression predicting student-perceived helpfulness, engagement-based metrics were the most robust predictors. Both RelScore (β = 0.420) and SuccScore (β = 0.187) were positively and significantly associated with helpfulness and remained stable in the combined model, whereas most pedagogical dimensions showed weak or inconsistent associations. Providing guidance was positively linked to helpfulness (β = 0.349), while merely identifying a mistake was negatively associated (β = −0.524).

Open Questions

• Can behavioral evaluation be automated at scale across different tutoring domains beyond programming education?
• How do behavioral metrics correlate with long-term learning outcomes vs. short-term perception?
• What is the optimal balance between pedagogical and behavioral evaluation weighting?

What this means for practice

• Designers. Evaluate deployed tutors on what students do with the feedback, not only on rubric quality: two tutors that scored similarly across eight pedagogical dimensions differed by 9–21 percentage points in feedback relevance on every assignment.
• Designers. Make feedback students will act on a first-class design objective — misconception-aware prompting raised uptake here — and accept the trade-off, since the misconception tutor reduced answer revealing (99.65 vs. 93.95 DAMR) at some cost to immediate actionability.
• Researchers. Interpret success scores before drawing conclusions: feedback marked undesired on the revealing-answer dimension achieved higher correct-application rates than desired feedback (79.4% vs. 53.0% for MisconceptionTutor), consistent with students copying a revealed answer rather than solving the problem.
• Researchers. Report engagement metrics alongside pedagogical ones, because RelScore (β = 0.420) and SuccScore (β = 0.187) predicted students' perceived helpfulness more consistently than most pedagogical dimensions did (guidance β = 0.349; mistake identification β = −0.524).
• Instructors. Define a directional success criterion for your own tutoring context — in dialog-based tutoring, whether a student's next response moves toward the desired understanding — so the framework transfers beyond code edits.

Limitations

• The two tutors were deployed in different semesters (BaselineTutor in Fall 2024, MisconceptionTutor in Fall 2025), so population and cohort differences may confound the observed effects; the authors plan randomized A/B deployment within a single semester to isolate tutor effects.
• The data come from a single course at one university (CS61A at UC Berkeley, roughly 1,000 students per semester) and only from programming, so transfer depends on defining a directional success criterion in the new domain.
• RelScore and SuccScore measure alignment between feedback and code edits rather than whether a student read the feedback, and large-scale rewrites or partially adopted suggestions complicate attribution.
• The metrics capture immediate feedback uptake only, not longer-term learning, and the tone and humanness pedagogy dimensions had too few undesired-feedback cases (n < 15) to be analyzed at all.

Citation

Niousha, R., Smith, S.B., Akram, B., Brusilovsky, P., Hellas, A., Leinonen, J., DeNero, J., & Norouzi, N. (2026). The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness