ValiAssist: Scaling LLM-generated Programming Feedback Validation by Demonstration

Synthesis: Large language models can now produce programming Feedback at scale, but instructors still must check whether each generated message fits the student's code and their own teaching strategy. ValiAssist reframes that check as end-user programming: an instructor's revision of a message becomes a demonstration, and the system infers a reusable when-then criterion, in a closed DSL of four predicates and four actions, that can be inspected and applied across the corpus. In a within-subjects lab study with 16 instructors, feedback validated under ValiAssist satisfied 91% of the instructors' own criteria against 57% under a natural-language baseline, and instructors repaired seeded criterion problems with 77% versus 32% accuracy. Logs show they did not specify once and walk away: they re-ran validation 49 times against the baseline's 26, editing criteria between runs. The paper's claim is that validation by demonstration turns judgments instructors already know how to make on individual cases into reusable logic, while clarifying where those judgments generalize — human-in-the-loop validation as progressive specification.

Key Findings

1. Revision-based criteria produced better-aligned feedback. Feedback validated under ValiAssist satisfied a mean of 91% of the instructors' own criteria; the natural-language baseline reached 57% (p < .01).
2. Correction accuracy more than doubled. Given criteria with seeded execution problems, instructors using ValiAssist fixed them with 77% accuracy versus 32% (p < .0001), and reported higher confidence (5.19 vs. 4.06).
3. Criteria covered more distinct pedagogical intents. Instructors' criteria spanned more unique intents under ValiAssist (μ = 2.56 vs. 2.00, p < .05); the count of criteria created did not differ significantly (3.00 vs. 3.13).
4. Structured criteria invited active maintenance. In the validation sub-task, instructors made far more criterion edits — new criteria, paraphrases, added examples — under ValiAssist than under the baseline (μ = 2.75 vs. 0.44, p < .001).
5. Scaling happened through repeated run-review cycles. Participants re-ran validation 49 times under ValiAssist versus 26 under the baseline, reviewing batches of about 6 criteria-proposed revisions rather than about 10.
6. Inference stayed competitive on matching; the baseline was far more liberal. Over 240 test cases, ValiAssist scored accuracy 0.68, precision 0.72 and recall 0.69; the baseline, given ground-truth intents, scored 0.65, 0.61 and 0.99.
7. Instructors judged messages against the submissions they came with. In a preliminary observation of 178 validation episodes, 57.87% interleaved between code and feedback and 69.66% referred to the submission, so criteria must range over both.

What the study actually compared

ValiAssist was evaluated in a within-subjects lab study with 16 instructors (8 male, 7 female, 1 non-binary) at a local university, all with programming (μ = 3.94 years) and teaching experience (μ = 2.38 years). Participants used the system under two conditions on two authentic introductory-level Python exercises. The baseline replicated the common LLM-as-a-judge practice of authoring criteria in pure natural language; because both conditions allowed criteria to be created from a demonstration, the comparison isolates the representation rather than the act. Sessions ran about 90 minutes with three timed sub-tasks: creating criteria from 5 code-feedback pairs, applying them to 35 pairs, and repairing two criteria with seeded execution problems, 4 pairs each. Alignment was not a general judgment of feedback quality: two researchers sampled 88 validated feedback instances (up to 3 per condition per participant) and measured the percentage of that instructor's own criteria each one satisfied.

Structured criteria as an inspectable artifact

ValiAssist represents a criterion as a when-then rule in a small DSL: conditions are built from four predicates (has, lacks, style, length) and the action from add, remove, revise, or replace. Arguments are either open-ended semantic labels the model judges at execution — for example "full solution" — or exact text matched programmatically, so an instructor can state which they mean. The closedness is the design bet: an instructor can read the whole language, edit a rule directly, and compare two rules for redundancy or conflict. The system grounds each criterion in its source code-feedback pair, highlights matched spans, and reports which part of a condition fired; when a pending criterion duplicates, refines, or contradicts a confirmed one, a dialog explains the overlap before acceptance. That makes Evaluative Judgment inspectable and pushes instructors toward feedback literacy, since the artifact to read and repair is now their own rule rather than the model's output.

Validation as progressive specification

The paper's most consequential finding is about rhythm. Under both conditions, hands-on revision of individual messages was rare — fewer than two revisions per participant in the validation sub-task — and instructors scaled by running criteria and reviewing the revisions those runs proposed. Under ValiAssist that loop tightened: 49 runs versus 26, in smaller batches of about 6 versus 10, with criterion edits interleaved between runs. One participant's timeline shows five runs, each followed by a burst of accepted revisions (18 accepted, 2 rejected); under the baseline the same instructor ran three later sweeps and accepted only 6 propagated revisions. The authors read this as demonstration serving as a starting point rather than a complete specification: when behavior depends on heterogeneous contextual evidence, the hard problem is helping users discover the boundaries of an inferred rule. That positions Human AI Collaboration as co-validation and shifts the instructor's Teaching from revising feedback to supervising criteria.

The risk the system targets

The authors argue that the sharpest risk in LLM-assisted validation is not inefficiency but miscalibrated trust. In the baseline condition, some participants treated natural-language criteria and model judgments as reliable after a first pass even when the validated feedback still violated their own standards: one created 5 criteria in a "vibing" way, never revisited them, and accepted revisions that broke most of them; another reached very low validation accuracy yet still expected the approach to "be very powerful if it could work." The paper describes two Guardrails against that drift: structured criteria make expectations inspectable and revisable, and demonstration grounds rule creation in concrete revision work instead of abstract prompt authoring. Together they keep the instructor inside the loop, so that assistance scaffolds judgment rather than becoming unchecked delegation.

What this means for practice

• Instructors. Treat your own revision as the specification: revise one misaligned message in place, read the criterion the system infers before accepting it, and re-run after every edit instead of approving a first pass — the study's alignment gains came with 49 re-runs against the baseline's 26 and criterion edits between them.
• Educational technology developers. Keep the rule vocabulary closed and readable, show per-condition matching status with evidence spans, and flag pending rules for duplication or conflict on acceptance; the 77% versus 32% correction gap appeared with inspectable execution, at no measured increase in perceived workload.
• Researchers. Log every inferred validation rule, its acceptance, and the cases it later misclassified: progressive specification is a claim about how criteria stabilize over time, while this study measured alignment inside a single lab session.

Limitations

• Evaluation ran in a lab setting with authentic student programming data rather than a genuine grading environment, so the pressure of misleading real students' learning progress was reduced.
• Each condition was tested for about 30 minutes inside roughly 90-minute sessions, so long-term reuse of criteria across assignments was not examined.
• The authors ran no deployment across a full term or a class of hundreds of students, leaving open whether leverage holds as criteria sets grow over weeks or when submissions arrive with patterns the demonstrations did not cover.
• The sample was 16 instructors at a single local university, and both exercises were introductory-level Python (function definition, for loops, conditional statements, list operations).
• Inference and execution relied on OpenAI's GPT-4o (temperature 0) and text-embedding-3-small embeddings with no data-collection window reported; the pipeline is nondeterministic (multi-run κ = 0.81 across 2085 data points).
• Alignment was scored against each instructor's own criteria (88 sampled instances), not general pedagogical soundness, which the study explicitly did not measure; the technical evaluation also gave the natural-language baseline ground-truth intents while ValiAssist inferred criteria from demonstration.

Citation

Tang, Abulikemu, Gao, and Chen (2026). ValiAssist: Scaling LLM-generated Programming Feedback Validation by Demonstration. arXiv preprint arXiv:2610.27248 (cs.HC).