PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

Synthesis: Roy and colleagues (2026) argue that a reusable rubric is not documentation wrapped around an LLM judge but the measurement specification itself: the same judge conditioned on two different rubrics is two different evaluators. PReMISE treats that rubric-conditioned judge as the object of study and audits any rubric set along four axes — structural adequacy, reliability, preference fit, and adversarial robustness — across four public human-preference corpora, six rubric sources, and three open-weight judges. No rubric source leads every axis, six different methods lead at least one, and high inter-judge agreement does not imply low exploitability: judges can agree on a flawed interpretation while rubric wording still admits responses that score well by violating the construct. Two repair operations move a rubric on the axes that fail — preference-rank selection raises paired-response judge accuracy from 65.0% to 68.6%, and reliability-constrained refinement cuts the rate at which exploit responses earn high scores from 46.4% to 36.0% at a small cost in agreement (α = .531 → .519). The study was not run in an educational setting; the bridge to AI grading is a mechanism inference, drawn out below.

Key Findings

1. The rubric, not just the model, determines the score. A fixed judge paired with two rubrics produces two different measurements, and a vague rubric — the paper's example asks for "helpful and factual" — can reward a polished answer that invents a policy detail, because the rubric gives the judge no criterion that the invention violates.
2. No rubric source is simultaneously reliable, preference-predictive, and robust. Across the four-axis audit, six different methods lead at least one column: AgentEval leads inter-judge agreement (α = .58), paraphrase stability (ρ = .70), and per-prompt applicability (.82); Auto-Rubric leads robustness (VFR = .27) and judge accuracy (68.7); AutoRule leads effective dimensionality (18); PReMISE leads structural adequacy (4.45) and source specificity (ΔApp = +.09).
3. Agreement is not robustness. High inter-rater agreement can coexist with high exploitability, because judges can share a systematic misreading; the paper therefore reports Krippendorff's α, winner flip rate, and paraphrase stability as stability checks rather than as evidence of validity.
4. PReMISE is the only rubric set scoring non-trivially on applicability, specificity, and effective dimensionality at once. Its discovery pipeline mines policy-level criteria from pairwise human preferences and consolidates them, rather than asking a model to invent criteria prompt by prompt.
5. Preference-rank selection lifts accuracy from 65.0% to 68.6%. Reordering discovered criteria by the fraction of preference pairs on which the chosen response scores higher raises held-out cross-source, cross-judge accuracy from 65.0% to 68.6%, competitive with the strongest discovery baselines and leading on two of three judges.
6. Reliability-constrained refinement cuts exploit success from 46.4% to 36.0%. Editing criterion text through a fixed operator library (applicability gates, do-not-reward clauses, maximum-score caps, decomposition, abstract-to-observable rewrites, false-positive anchors, non-compensation rules) lowers Verified Fool Rate from 46.4% to 36.0% while inter-judge agreement moves only from .531 to .519.

A rubric is the measurement, not the wrapper

The paper's framing is deliberately deflationary about model choice. Given a fixed auto-rater J, a rubric r induces an evaluator that scores a response x to prompt p; downstream pipelines consume that rubric-conditioned evaluator, not the rubric text alone. Two rubric sets can reach similar accuracy through different anatomy — one broad and generic, another narrow and specific, a third spread across many independent criteria — so accuracy without the structural audit hides which instrument produced the number.

The illustrative failure is not an exotic attack. A rubric asking for "helpful, factual, and clear" can lead a judge to score a fluent answer highly even when it invents a return window that contradicts the stated policy: the bad answer is one problem, and the deeper measurement problem is that the induced judge had no criterion under which the invention counted against the response. That is the sense in which the rubric is a specification: editing it changes what the judge can see.

Four audit axes, and no source wins them all

PReMISE places six rubric sources on common footing. Structural adequacy asks whether a rubric is well-formed — atomic, internally consistent, observable, operationalizable, and unambiguous in scope. Reliability probes invariance three ways: inter-judge agreement (Krippendorff's α), intra-judge winner flip rate across stochastic re-runs, and paraphrase stability under semantically equivalent rewrites. Preference fit asks whether the rubric adds information over a no-rubric prior, measured both structurally (per-prompt applicability, source specificity, effective dimensionality) and empirically (rubric-conditioned judge accuracy). Adversarial robustness asks how readily a high score can be obtained while violating the intended construct, using an attacker–judge–verifier protocol: an attacker sees the rubric text and writes a response designed to satisfy its wording while breaking its intent, the judge scores it, and a verifier blind to both labels high-scoring candidates as exploits.

The audit's headline is a negative result with practical force: no source leads every axis. AgentEval wins agreement and stability, Auto-Rubric wins robustness and raw accuracy, AutoRule wins dimensionality, and PReMISE wins structural adequacy and specificity. The dissociation between reliability and robustness is the point — a rubric that produces consistent scores across judges can still be easy to game, and a rubric that resists gaming can be less consistent. The authors are also explicit that LLM-mediated audit metrics are scalable diagnostics, not human validation: shared model biases can produce high agreement on a common error.

Two repairs, and what they buy

Both repairs are editable-rubric operations rather than model training. Preference-rank selection keeps criterion text unchanged but re-selects and re-orders the top K discovered criteria by a preference-fit score — the fraction of human-preference pairs on which the chosen response scores higher — so criteria that discriminate between responses but point in the wrong direction are demoted. It raises accuracy from 65.0% to 68.6% as a mean across four sources, three judges, and two prompt templates.

Reliability-constrained refinement edits criterion wording through a fixed operator library, using failure packets that mix true positives, true negatives, false accepts, false rejects, borderline and unstable cases. Candidate revisions must satisfy a structural budget (total ≤ 150 words, at most two anti-gaming clauses, a single applicability gate, prompt-agnostic) and are accepted only when they lower exploit rate subject to an agreement floor. The result is a Pareto frontier over agreement and robustness rather than a single "best" rubric: operators get distinct operating points, and when no candidate satisfies the rule the criterion is left unchanged. The exploit-rate drop from 46.4% to 36.0% is real but partial — more than a third of verified exploit attempts still earn high scores.

What this could mean for AI grading in education

The paper never tests a classroom, an essay, or a course rubric; its corpora are general preference datasets (UltraFeedback, Arena-Expert-5K, HelpSteer3, HH-RLHF) and its rubric sources are model-policy specifications. The bridge is mechanical rather than empirical: an AI grader conditioned on a marking rubric is the same kind of object PReMISE audits, and the failure it names — a vague criterion that a fluent, non-compliant response can satisfy — is a familiar assessment-design problem. Two implications follow only as hypotheses that would need testing in an educational setting. First, institutions that validate an AI grader by comparing its marks with human markers are checking one axis (reliability/preference fit) while leaving adversarial robustness untested, although the education version of an exploit — a response written to satisfy rubric wording rather than the construct — is exactly what a grade-motivated student would produce. Second, rubric text becomes a versioned part of the assessment instrument: changing "clear and well-organized" to an observable criterion changes the measurement, so a rubric revision is a revalidation event, not copy-editing.

What this means for practice

• Treat the rubric as the instrument and version it. If the same model is deployed with a revised rubric, the change is a measurement change; record the rubric text, date, and judge configuration alongside the marks.
• Test exploitability as a separate axis from agreement. Writing a handful of responses designed to satisfy the rubric's wording while violating its intent — and checking whether they score highly — probes something that inter-rater agreement cannot.
• Write grading criteria that name observable behaviors. The refinement operators are a usable checklist: applicability gates, explicit do-not-reward clauses, maximum-score caps where appropriate, decomposition into sub-criteria, and false-positive anchors drawn from past over-crediting.
• Do not assume a larger judge removes the problem. On Qwen3-32B, PReMISE with preference repair ranked below the no-rubric baseline; the paper treats judge capacity as a live constraint, and the same caution applies to smaller models schools can afford to run.
• Read an agreement statistic as one check, not a warrant. The paper's central measurement claim — agreement is not validity, and it is not robustness — is the discipline an AI-grading pilot needs before it treats a kappa as the case for deployment.

Limitations

• The evidence is non-educational. No classroom, student work, course rubric, or learning outcome was studied; the education bridge above is an inference from a shared measurement mechanism, not a finding, and any transfer would need local validation with authentic student responses.
• PReMISE targets policy-level rubrics audited against pairwise preference data. Rubrics meant to enforce objectives that should override preferences — safety floors or hard policy constraints — are outside its scope, and educational rubrics often combine both preference-aligned quality and non-negotiable correctness.
• The adversarial audit measures a lower bound on exploitability under one attacker–judge–verifier protocol; stronger adversaries may find exploits the study missed, and weaker ones may miss exploits it surfaced, so the 36.0% figure is protocol-specific rather than a property of the rubric.
• All audit metrics are LLM-mediated and therefore susceptible to systematic bias shared across model families; agreement, flip rate, and paraphrase stability are stability checks, not human-validated evidence of construct validity.
• Judge capacity matters: the repair pipeline's advantage disappeared on Qwen3-32B, and adapting policy-rubric methods to smaller models remains open.
• The paper reports no educational effect, no student outcome, and no cost analysis; it establishes a measurement framework and an audit result, not a case that rubric auditing improves grading in practice.

Citation

Roy, S., Pujari, R., Kumarage, T., Peris, C., Gupta, R., Rumshisky, A., Natarajan, P., & Saligrama, V. (2026). PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges. arXiv preprint.