EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics

Synthesis: EduFair-Bench is the first Benchmark to score pedagogical quality and demographic fairness jointly, pairing a filtered mathematics/physics/chemistry question bank with a fixed simulated student across nine demographic levels. Auditing five tutors from 7B to 70B, the authors find that model capability and demographic fairness are largely orthogonal — the smallest tutor is the most consistent while four stronger tutors show wide gaps with no capability-to-fairness ordering. Pedagogy-specific RL redistributes rather than removes bias, and language- and immigration-related cues produce larger disparities than gender or SES. Because name-only cues and deliberately mismatched demographic statements both amplify the gaps, the audit implicates tutor-side demographic processing as a major mechanism of biased tutoring behavior.

Key Findings

1. EduFair-Bench evaluates pedagogical fairness across nine demographic levels spanning four dimensions (gender, immigration background, first language, SES) using a five-metric turn-level taxonomy plus four conversation-level metrics, judged by a human-validated LLM judge.
2. Biases concentrate in different behaviors per domain: in mathematics the largest effects fall on step_scaffolded (up to |r| = 0.144) and actionability (up to 0.169), while in chemistry and physics corrective_tone exceeds |r| = 0.10 for all five tutors (chemistry 0.102–0.168; physics 0.129–0.294).
3. Language and immigration cues drive the largest gaps: the three biggest cells are First Language for DeepSeek-R1-70B (0.281) and GPT-5-mini (0.206), and Immigration for DeepSeek-R1-70B (0.216), with gender and SES disparities smaller and more diffuse.
4. Capability does not predict fairness: Qwen2.5-7B exceeds the |r| ≥ 0.10 threshold in 7 of 12 domain–dimension cells while LLaMA-3.1-8B exceeds it only once, and DeepSeek-R1-70B simultaneously shows the largest language and immigration gaps yet the smallest gender bias (0.022 in chemistry).
5. Pedagogy-specific RL redistributes rather than removes bias — relative to its Qwen2.5-7B backbone, TutorRL-7B suppresses corrective_tone bias (chemistry 0.159 → 0.104; physics 0.294 → 0.163) but raises mistake_recognize bias (math 0.102 → 0.165; physics 0.150 → 0.212), while amplifying First Language disparities in all domains (math 0.069 → 0.144; chemistry 0.114 → 0.177; physics 0.081 → 0.137).
6. Names alone suffice to trigger pedagogical bias, especially in physics (mean pedagogical |r| ≈ 0.142 versus ≈ 0.074 for mathematics and ≈ 0.071 for chemistry), and combined gender+ethnicity name cues reach 0.205 for LLaMA-3.1-8B and 0.185 for Qwen2.5-7B.
7. Mismatched demographic conditioning amplifies divergence: for 11 of 15 (tutor, domain) cells the Wrong condition exceeds Correct, with Qwen2.5-7B/math rising 0.073 → 0.238, TutorRL-7B/math 0.066 → 0.154, DeepSeek-R1-70B/physics 0.043 → 0.169, and GPT-5-mini/physics 0.165 → 0.290.
8. The LLM judge (gpt-oss-120b) was validated against three annotators on 180 tutor turns, with pre-discussion Krippendorff's α from 0.81 to 1.00 and judge–human agreement from 0.79 to 1.00 across dimensions, after four rubric-refinement workshops.

Method: A Paired Demographic Audit

EduFair-Bench is built on a multi-domain question bank drawn from two source corpora. Mathematics items come from SocraTeach, filtered on reference dialogue signals (at least four student–tutor exchanges, at least two annotated student errors, at least one correct or partially correct response after the final error, and no explicit ground-truth disclosure), reducing 35,151 items to 907 questions. Physics and chemistry items come from SciQ, where a keyword-seeded TF-IDF classifier separates domains and retains items classified with confidence ≥ 0.65, sampling up to 500 questions per domain from 11,679 items. The bank therefore anchors conversations but never supplies them.

Each dialogue is generated by pairing one of five Large Language Models (LLMs) tutors with a single fixed simulated student — LLaMA-3.1-8B-Instruct, decoded at temperature 0 — so that within a paired comparison question and student are held constant and any difference is attributable to the tutor's response to the demographic signal. The tutors span model families, scales and training paradigms: LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct (instruction-tuned), TutorRL-7B (pedagogy-tuned with RL), DeepSeek-R1-Distill-Llama-70B (reasoning-tuned), and GPT-5-mini (proprietary frontier). Dialogues alternate turns until the tutor reveals or confirms the correct answer, capped at T_max = 10 turns, against reference dialogues that span 4–7 turns. The experimental conditions vary not only which signal is present but how it is delivered: Explicit (a direct demographic statement), Implicit (a demographically connoted name), and Opposite (a 2 × 2 design in which the student's self-stated gender and the tutor's perceived gender are manipulated independently).

Scoring uses a taxonomy adapted from Maurya et al.'s eight-dimension pedagogical evaluation taxonomy with three changes: mistake_recognize merges identification and location, step_scaffolded and actionability remain separate because pilot annotations showed they vary independently, and Human-likeness and Coherence-with-context are dropped. Four pedagogical functions — Diagnose (mistake_recognize), Reveal (answer_leakage), Next-Step (step_scaffolded, actionability) and Communicate (corrective_tone) — structure the five turn-level metrics, with −1 assigned when a dimension is not applicable. Judge independence is buttressed by four conversation-level metrics (tutor_turns, tutor_avg_word, tutor_question_rate, tutor_student_word_ratio) extracted deterministically from transcripts, which provide triangulation that does not depend on the judge at all.

Evidence: Where Bias Emerges

Bias profiles differ sharply across domains. In mathematics the strongest effects concentrate on proactive behaviors, particularly step_scaffolded (up to |r| = 0.144) and actionability (up to 0.169), while corrective_tone stays uniformly low (0.022–0.073). The pattern reverses in chemistry and physics, where scaffolding and actionability mostly remain below 0.10 but corrective_tone exceeds it for every model (chemistry 0.102–0.168; physics 0.129–0.294). Demographic signals, in other words, primarily affect how a tutor scaffolds mathematics but how it corrects errors in the sciences.

Across demographic dimensions, language and immigration cues dominate. Averaged over the five turn-level metrics, the largest Explicit cells are First Language for DeepSeek-R1-70B (0.281) and GPT-5-mini (0.206) in physics, and Immigration for DeepSeek-R1-70B (0.216). Gender and SES disparities are smaller and more diffuse. Sensitivity to demographic cues is itself a model property: Qwen2.5-7B exceeds the |r| ≥ 0.10 threshold in 7 of 12 domain–dimension cells, whereas LLaMA-3.1-8B exceeds it only once, and DeepSeek-R1-70B pairs the largest language and immigration gaps with the smallest gender bias (0.022 in chemistry). No model exceeds 0.10 on all four demographic dimensions within any domain.

The clearest test of whether capability buys fairness comes from comparing TutorRL-7B with its Qwen2.5-7B backbone. Pedagogy-specific RL suppresses corrective_tone bias (chemistry 0.159 → 0.104; physics 0.294 → 0.163) and answer_leakage bias (chemistry 0.121 → 0.045; physics 0.067 → 0.047), but increases mistake_recognize bias (mathematics 0.102 → 0.165; physics 0.150 → 0.212) and worsens First Language disparities in every domain (mathematics 0.069 → 0.144; chemistry 0.114 → 0.177; physics 0.081 → 0.137). Optimizing for named pedagogical behaviors moves demographic inequality around rather than eliminating it — a finding that bears directly on evaluation practice, since a benchmark scoring only aggregate teaching quality would miss the redistribution entirely.

Ablations: Names, Mismatch, and the Tutor-Side Signal

The Implicit condition isolates whether demographic information must be stated outright. It need not be. Name-mediated pedagogical bias is substantially larger in physics (mean ≈ 0.142) than in mathematics (≈ 0.074) or chemistry (≈ 0.071), and rows combining gender and ethnicity cues ("Both") produce the strongest effects for most tutors, reaching 0.205 for LLaMA-3.1-8B and 0.185 for Qwen2.5-7B. Joint cues therefore generally induce larger disparities than either signal alone, while conversation-level effects stay comparatively small — name-mediated bias primarily reshapes pedagogical behavior rather than dialogue structure. Models also respond differently to cue channel: LLaMA-3.1-8B shows minimal Explicit bias yet reaches pedagogical |r| = 0.205 under Implicit, exceeding all Implicit values of GPT-5-mini, whereas GPT-5-mini is consistently more sensitive to explicit statements.

The Opposite condition asks whether the residual gender signal is tutor-driven or an artifact of the student simulator. Because the student's self-stated gender and the tutor's perceived gender are varied orthogonally, matched (Correct) and mismatched (Wrong) cells decompose the gap into tutor effects, simulator effects and their interaction. Mismatch amplifies disparities in 11 of 15 (tutor, domain) cells: Qwen2.5-7B/mathematics 0.073 → 0.238, TutorRL-7B/mathematics 0.066 → 0.154, DeepSeek-R1-70B/physics 0.043 → 0.169, and GPT-5-mini/physics 0.165 → 0.290. Only four cells reverse the pattern, and all reversals are small (≤ 0.043), spanning all three domains, which suggests model-dependent rather than domain-dependent behavior. Contradictory demographic signals thus amplify divergence beyond what the student persona alone induces, implicating tutor-side demographic processing as a major source of bias.

What this means for practice

• Designers. Audit each tutor for demographic bias before classroom deployment, because capability does not predict fairness — one model exceeded the |r| ≥ 0.10 threshold in 7 of 12 domain–dimension cells while another exceeded it only once — and passing the benchmark's thresholds is not a certificate of equitable treatment.
• Designers. Treat pedagogy-specific RL as redistributing bias rather than removing it: TutorRL-7B reduced corrective_tone bias (chemistry 0.159 → 0.104) but increased mistake_recognize bias (mathematics 0.102 → 0.165).
• Designers. Test tutors with name-only demographic cues as well as explicit statements, since names alone produced a mean pedagogical effect of about 0.142 in physics, and score the audit with paired contrasts that hold question difficulty constant — at least 15 paired questions per contrast — recording a fairness violation when the confidence interval excludes zero at |r| ≥ 0.10.
• Instructors. Attend first to language and immigration fairness, where the largest gaps appeared (First Language 0.281 and Immigration 0.216 for DeepSeek-R1-70B), rather than to gender or SES alone.
• Researchers. Validate audit findings against human-subjects studies before making strong classroom claims, because simulated students diverge from real learners in error patterns, persistence and Help-Seeking.

Limitations

• Every dialogue uses one fixed simulated student (LLaMA-3.1-8B-Instruct), and LLM-simulated students are known to diverge from real learners in error patterns, persistence and help-seeking, so the patterns need human-subjects validation.
• No intersectional analysis: the four dimensions are treated as independent axes, and the Implicit condition covers only gender and ethnicity jointly, while bias across protected attributes is known to interact non-additively.
• The demographics are coarse and often binary ("native-born"/"immigrant", "English L1"/"non-English L1"), which the authors acknowledge erases within-group heterogeneity.
• GPT-5-mini was evaluated on a fixed stratified 30% subsample per domain (272/150/150 items) rather than the full question bank, so its results are not directly comparable to the four open-weight tutors run on the full bank.

Citation

Jiaxu Zhao, Bahar Radmehr, Fares Fawzi, Tanya Nazaretsky, and Tanja Käser (2026). EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics. arXiv preprint.