MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

Synthesis: MisconceptDialogue is a fully automated benchmarking framework that runs the complete detect–diagnose–remediate pipeline on five open-weight LLM tutors (7B–14B) interacting with LLM-simulated students across over 5,000 dialogues in mathematics. Its headline result is a measurement warning: on identical dialogues, benchmark-style exact-match scoring reports 42.1% Remediation Success Rate while a human-calibrated LLM-as-judge reports 81.3%, a 39.2 percentage-point spread larger than the combined effects of tutor model, teaching strategy, and supplied diagnostic information. The gap is not a single-judge artifact — four independent judges from three vendor families converge within 6 percentage points (Fleiss κ = 0.85), and 200 human annotations anchor it (κ = 0.74–0.91). On capability, structured pipeline information gave no detectable advantage over direct tutoring (p = 0.263), and a wrong-diagnosis control found little evidence that these tutors condition on the labels they are handed. Every capability claim is scoped to this simulated protocol and does not establish real-student learning.

Key Findings

1. The scoring criterion moves the number more than anything else. Strict match scored the same 2,250 dialogues at 42.1% and the human-calibrated LLM-as-judge at 81.3%, a 39.2 percentage-point gap that exceeds model, strategy, and pipeline-condition effects combined.
2. The gap is a property of the criterion, not the judge. Three main judges converge at Fleiss κ = 0.85 with pairwise Cohen κ = 0.78–0.93, a 3-judge majority vote of 78.2% sits within 3 percentage points of each judge, and GPT-5.4 tracks GPT-5-mini within 2.8 percentage points.
3. Context helps detection only on imbalanced data. Multi-turn context improved Macro F1 for all five models on BEA 2025 and Bridge, but all five lost F1 on mixed-label MathDial (Δ: −0.026 to −0.163), because predictions skew toward "incorrect" (mean 55.8%→79.7%).
4. Detection and diagnosis are different capabilities. Mistral-7B had the strongest mixed-label detection F1 (0.492) yet the lowest best-case diagnosis accuracy (≤0.255), while Qwen3.5-9B led diagnosis (0.545) with only moderate detection (0.438).
5. Pipeline structure did not measurably help. Progressive oracle ablation across six conditions found no significant pipeline-condition effect (GLMM χ² = 6.47, p = 0.263), with RSR ranging only from 76.6% to 81.3% and no significant pairwise differences.
6. Tutors did not reliably use the diagnosis they were given. Supplying oracle detection with a deliberately swapped same-type misconception yielded 80.8%, statistically indistinguishable from gold diagnosis (76.6%) or direct tutoring (78.5%) across three independent judges.
7. Misconception type, not strategy, drove outcomes. The LLM-as-judge showed a monotone ordering by type (conceptual 89.9% > procedural 80.9% > factual 73.2%), while the five remediation strategies clustered within 6.7 percentage points.

Framework and design

The framework separates three objects that tutoring evaluation often conflates: the D-G-R tutoring pipeline, the pipeline-information interventions controlling what the tutor is told, and the evaluation lenses applied to the resulting dialogues. The scenario bank instantiates 55 educator-validated middle-school algebra misconceptions from the Math Misconceptions Benchmark into 106 scenarios, 36 factual, 34 procedural, and 36 conceptual, each carrying a problem statement, an expected student response, a free-text label, and a held-out verification problem used to score Remediation Success Rate. Scenarios mix existing Benchmark items (53%) with GPT-5.4-generated variants (47%). Five remediation strategies were drawn from learning-science work on misconception and conceptual change: Counterexample, Analogy, Cognitive Conflict, Direct Instruction, and Socratic Questioning.

Simulation is a deliberate trade of ecological validity for experimental control. It holds the student fixed while tutor-side information varies across six paired pipeline conditions, permits a wrong-diagnosis control that could not responsibly be run on real struggling students, and scales to over 5,000 dialogues so that criterion effects carry paired statistics. A GPT-5-mini simulated student adopts the assigned misconception faithfully and completes 5 tutoring turns plus a verification problem. The tutor models are Qwen-2.5-7B, Mistral-7B, Llama-3.1-8B, Phi-4 (14B), and Qwen3.5-9B (9B), all open-weight and run locally on H100 GPUs.

The measurement result

The paper's central methodological move is to score identical dialogues with multiple criteria, decomposing measured success into what the tutor produces versus what the scorer credits. On 2,250 remediation dialogues, strict match (word-boundary regex) produced 40.2–44.4% RSR, numeric extraction with tolerance 44.4–47.1%, and the human-calibrated LLM-as-judge 77.8–84.4%. Because the dialogues are identical, the gap reflects what counts as correct, not changes in tutor behavior or answer revealing, which stayed at an Answer Reveal Rate of 6.0–8.7%. Paired McNemar testing on the identical dialogues confirms the gap (χ² = 765.7, p < 10⁻⁵⁰; 95% bootstrap CI: [36.6, 41.8] percentage points).

Cross-tabulating the criteria decomposes the 39.2 percentage-point gap into two components. Strict-fail/judge-pass disagreements total 949 dialogues, 42.2% of all dialogues: 23.4% (222) are also numeric-pass, exposing string mismatches on numerically correct answers, while 76.6% (727) are numeric-fail/judge-pass, capturing paraphrase tolerance and reasoning credit beyond exact-answer matching. Roughly three-quarters of the gap is therefore construct-credit difference, with the remaining quarter attributable to the surface-matching brittleness of Benchmark-style exact match. Strict match is not an arbitrary baseline: it mirrors the exact-match scoring that standard math NLP benchmarks apply to model outputs, so including it directly tests how much current practice would lower measured remediation estimates if transferred to tutoring dialogues.

Capability results and human calibration

Turning to capability, the paper reports a pipeline null. Progressive oracle ablation across five models and six conditions found no significant pipeline-condition effect (GLMM χ² = 6.47, p = 0.263). Variance decomposition explains why: scenario difficulty dominates (intraclass correlation, ICC = 0.206), while model and condition effects are negligible (ICC < 1%), so pipeline-information signals are smaller than scenario-level variation. The authors accept equivalence at δ = 10 percentage points for all 15 pairs but only three near-identical pairs at δ = 5 percentage points, so benefits around 5 percentage points remain possible.

A more direct control tests whether tutors condition on supplied diagnoses. Supplying oracle detection with an intentionally mismatched same-type misconception yields 80.8%, indistinguishable from gold diagnosis (76.6%; McNemar p = 0.078) or direct tutoring (78.5%; p = 0.315). Three independent judges agree on the wrong-versus-gold direction (Claude Haiku 4.5 +3.2, Gemini 3.1 Flash Lite +3.4, GPT-5-mini +4.2 percentage points), arguing against a single-judge artifact. The null is not a simple persuadability ceiling: profile-level RSR varies by 21.2 percentage points and persona-only RSR is 6.6% against 81.3% with tutoring, so the simulator is not a uniform recoverer.

Human calibration anchors the automated scores. Three judge-blinded annotators, each with at least a bachelor's degree and prior coursework or equivalent experience in benchmark-level mathematics, rated two quality-stratified subsets of 100 dialogues each on correct understanding, reasoning quality, answer revelation, and naturalness. Inter-annotator agreement was moderate (mean pairwise Cohen's κ = 0.51 clean, 0.55 borderline). All four LLM judges track human consensus closely, within 1–3 percentage points on clean cases and 2–7 percentage points on borderline cases, with item-level κ ranging 0.79–0.91 clean and 0.74–0.87 borderline. Gemini 3.1 Flash Lite, a Google model with no OpenAI lineage, reached the highest agreement on both subsets (κ = 0.905 clean, 0.872 borderline).

What this means for practice

• Researchers. Report tutoring success under at least two automated criteria, quantify their disagreement, and include a human baseline, because criterion alone shifted RSR by 39.2 percentage points while model, strategy, and judge identity each contributed only single-digit effects.
• Researchers. Audit the disagreement set directly rather than collapsing criteria into one preferred score, and treat the scoring criterion as a first-class object of study rather than an implementation detail.
• Educational technology developers. Do not import Benchmark-style exact match wholesale into dialogue tutoring systems; numeric extraction also stays far below understanding judgment, and Answer Reveal Rate of 6.0–8.7% shows tutors still give away answers.
• Instructors. Treat a simulated-student success rate as an upper bound on classroom effectiveness, since simulated students cannot fully capture diverse learners' reasoning, language backgrounds, or affective responses, and the authors recommend human expert review before autonomous deployment.
• Assessment designers. Score formative remediation for conceptual understanding, not answer strings, because 76.6% of the criterion gap comes from reasoning credit that exact match cannot see.

Limitations

• The study involves no real student participants and uses LLM-simulated students, so it bounds rather than establishes real-classroom effectiveness; simulated students cannot fully capture diverse learners' reasoning, language backgrounds, or affective responses.
• Domain and taxonomy scope is narrow: mathematics was chosen for relatively clear-cut answers, and the 55 educator-validated middle-school algebra misconceptions are a small slice of real curricula.
• Detection datasets are imbalanced — BEA 2025 (n = 300) and Bridge (n = 419) contain only incorrect responses while MathDial (n = 2,262) is 75%/25% correct/incorrect — and the Exp. 4 C4 detection rate of 96.8% is recall, not specificity.
• The 47% of remediation scenarios generated by GPT-5.4, combined with the OpenAI-family default judge, could inflate RSR through in-family coherence; the clean-set human RSR of 88% and cross-vendor replication bound but do not eliminate this risk.
• The Exp. 4 null is bounded rather than exact: TOST accepts equivalence at δ = 10 percentage points for all 15 pairs but only three at δ = 5 percentage points, with 80% power for effects ≥ 7 percentage points and 56% at 5 percentage points.
• GPT-5-mini doubles as simulated student and one of four LLM judges in Exp. 3–4, and the human study covers two n = 100 quality-stratified subsets with moderate inter-annotator agreement (κ = 0.51–0.55), so the wrong-diagnosis control tests conditioning under one presentation format only and broader annotation pools would tighten calibration.

Citation

Lee, W., Woo, Y., Kim, J., Gwon, H.-M., & Huang, J.-X. (2026). MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring. arXiv preprint.