When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI

Synthesis: MagicSchool's K-12 suite serves millions of teacher and student messages monthly, and its offline AI Ed Evaluation layer scores output on four dimensions: student safety, tone and instructional role, pedagogical value, and structural quality. As the program matured, false positives dominated the flags that route scarce analyst attention. This paper reports a production case study of an Large Language Models (LLMs)-as-judge panel: a second judge re-examining 140,000 failures from 2.5 million evaluations over six months failed to corroborate 94% of them. Three levers — unanimous-fail panels, per-evaluator judge-model choice, and softened rubrics — were paired with a 2,482-case synthetic Benchmark and a 570-case egregious set. Across 21 deployed evaluators, median activation reached 0.04% (16 of 21 at or below 0.2%; 8 at 0.00%) while egregious capture held at 100%, and per-flag precision rose from 0.6% to 48.9%. For an offline product-decision queue, analyst attention — not user exposure — is the binding constraint.

Key Findings

1. EvalEye evaluates a K–12 suite along four dimensions — student safety, tone and instructional role, pedagogical value, structural output quality — offline after generation rather than blocking the user path.
2. False positives dominated the review queue: a second judge re-examining 140,000 failures from 2.5 million evaluations over six months could not corroborate 94% of them.
3. Three levers drove the fix: unanimous-fail panels of up to five repeated runs, per-evaluator judge-model choice across Sonnet 4.5, GPT-5.4, and Haiku 4.5, and softened rubrics.
4. Across 21 deployed evaluators, median benchmark activation fell to 0.04% (16 of 21 at or below 0.2%; 8 at 0.00%), against a baseline single-judge median of 1.9%.
5. Egregious-failure capture reached 100% for all 21 evaluators on a 570-case severe set, up from a per-evaluator median of 100% (range 73.3–100%) before calibration.
6. Per-flag precision rose from 0.6% (24 true positives among 4,002 flags) to 48.9% (43 among 88), a 99% cut in confirmed false-positive flags.

The false-positive flood

The suite includes lesson-planning tools, classroom-material generators, teacher and student chatbots, and writing-feedback products. EvalEye is one layer among several defenses: real-time moderation runs upstream, alongside hard-coded safety reminders, conversation constraints, prompting Guardrails, and human-review workflows. Each evaluator targets a single failure mode and returns a binary pass/fail verdict plus a rationale, feeding dashboards, launch checks, and analyst review queues. When most flags do not warrant a change, attention spreads across cases that do not need it. In one incident, a vendor upgrade to an evaluator's judge model lowered pass rates on Factual accuracy from 97% to 11% in a single day, and confirming that the drop was a measurement artifact rather than a real regression cost real investigation time.

Three calibration levers

A panel is k repeated runs of one judge model with one rubric, capped at five judges. The primary verdict is fail only if all k runs fail, and the first passing verdict short-circuits the panel. Judge-model choice was per evaluator: Sonnet 4.5 for semantic judgments about tone, writing craft, and pedagogy; GPT-5.4 for structural and mathematical verification; Haiku 4.5 for one lower-cost evaluator. Rubric softening applied require unambiguity (if the violation is a judgment call, pass), require pervasiveness (one local instance is not enough), and simplify checklists (judge whether the response as a whole shows the failure).

What the calibration bought

Median post-calibration activation was 0.04%, against a baseline single-judge mean of 7.7% and median of 1.9%. The highest remaining rates were Completeness at 0.81%, Well-formedness at 0.64%, and Truncation at 0.56% — and these were largely real: manual inspection showed the 20 Completeness failures were false positives, while the 16 Well-formedness and 14 Truncation cases were actual violations, no longer buried. At the case level, the distribution shifted from Educational value (1,839 to 13 cases) toward No failure (396 to 2,410), with cases flagged at any non-trivial category dropping from 2,086 to 72. Shallow was the largest single change, falling from 54.71% to 0.12% activation with full capture (29 of 29).

Lessons that carry over

For a mature LLM-as-judge suite the dominant failure mode is false positives: real failures become rare, and a false-positive rate as low as 1% swamps them. The signal must be actionable, tied to failures that would drive a product change, rather than a large volume of borderline flags. Prompts need to soften but still have teeth, which is why a standard benchmark is paired with severe cases that should always fail. Ensembles help on the critical cases, and short-circuiting keeps their cost close to a single judge call on traffic that would pass anyway. Each evaluator solves a different problem, so a uniform strategy would not come close to these gains.

What this means for practice

• Instructors. Treat an AI quality or safety flag as a prompt to exercise Evaluative Judgment: read the rationale and check whether the response actually exhibits the claimed failure.
• Edtech developers. Track activation rate and severe-case capture as a pair; activation falls trivially when a rubric is loosened until it stops firing.
• Assessment designers. Ground-truth a sample of flags before trusting aggregate rates; per-flag precision here was 0.6% before calibration.
• Product and safety leaders. Budget analyst attention as the binding resource and re-audit after any vendor model upgrade; one swap dropped a Factual accuracy pass rate from 97% to 11% overnight.

Limitations

• The 2,482-case benchmark is the same pool used to compare candidate configurations, with no held-out split, so reported rates are in-sample.
• Both pools are synthetic rather than drawn from production traffic and may underrepresent contested or culturally specific failures a production stream would surface.
• Egregious capture covers only a deliberately severe slice: denominators range from n = 12 to n = 79, with a 95% Clopper–Pearson lower bound of 73.5% at n = 12 and 95.4% at n = 79.

Citation

Rohlfs, C., Vergara Bosse, R., Hopper, P. R., O'Keefe, K., & Russell, P. (2026). When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI. arXiv preprint.