EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics

Synthesis: EduFair-Bench is the first Benchmark to score pedagogical quality and demographic fairness jointly, pairing a filtered mathematics/physics/chemistry question bank with a fixed simulated student across nine demographic levels. Auditing five tutors from 7B to 70B, the authors find that model capability and demographic fairness are largely orthogonal — the smallest tutor is the most consistent while four stronger tutors show wide gaps with no capability-to-fairness ordering. Pedagogy-specific RL redistributes rather than removes bias, and language- and immigration-related cues produce larger disparities than gender or SES. Because name-only cues and deliberately mismatched demographic statements both amplify the gaps, the audit implicates tutor-side demographic processing as a major mechanism of biased tutoring behavior.

Key Findings

1. EduFair-Bench evaluates pedagogical fairness across nine demographic levels spanning four dimensions (gender, immigration background, first language, SES) using a five-metric turn-level taxonomy plus four conversation-level metrics, judged by a human-validated LLM judge.
2. Biases concentrate in different behaviors per domain: in mathematics the largest effects fall on step_scaffolded (up to |r| = 0.144) and actionability (up to 0.169), while in chemistry and physics corrective_tone exceeds |r| = 0.10 for all five tutors (chemistry 0.102–0.168; physics 0.129–0.294).
3. Language and immigration cues drive the largest gaps: the three biggest cells are First Language for DeepSeek-R1-70B (0.281) and GPT-5-mini (0.206), and Immigration for DeepSeek-R1-70B (0.216), with gender and SES disparities smaller and more diffuse.
4. Capability does not predict fairness: Qwen2.5-7B exceeds the |r| ≥ 0.10 threshold in 7 of 12 domain–dimension cells while LLaMA-3.1-8B exceeds it only once, and DeepSeek-R1-70B simultaneously shows the largest language and immigration gaps yet the smallest gender bias (0.022 in chemistry).
5. Pedagogy-specific RL redistributes rather than removes bias — relative to its Qwen2.5-7B backbone, TutorRL-7B suppresses corrective_tone bias (chemistry 0.159 → 0.104; physics 0.294 → 0.163) but raises mistake_recognize bias (math 0.102 → 0.165; physics 0.150 → 0.212), while amplifying First Language disparities in all domains (math 0.069 → 0.144; chemistry 0.114 → 0.177; physics 0.081 → 0.137).
6. Names alone suffice to trigger pedagogical bias, especially in physics (mean pedagogical |r| ≈ 0.142 versus ≈ 0.074 for mathematics and ≈ 0.071 for chemistry), and combined gender+ethnicity name cues reach 0.205 for LLaMA-3.1-8B and 0.185 for Qwen2.5-7B.
7. Mismatched demographic conditioning amplifies divergence: for 11 of 15 (tutor, domain) cells the Wrong condition exceeds Correct, with Qwen2.5-7B/math rising 0.073 → 0.238, TutorRL-7B/math 0.066 → 0.154, DeepSeek-R1-70B/physics 0.043 → 0.169, and GPT-5-mini/physics 0.165 → 0.290.
8. The LLM judge (gpt-oss-120b) was validated against three annotators on 180 tutor turns, with pre-discussion Krippendorff's α from 0.81 to 1.00 and judge–human agreement from 0.79 to 1.00 across dimensions, after four rubric-refinement workshops.

Method: A Paired Demographic Audit

EduFair-Bench is built on a multi-domain question bank drawn from two source corpora. Mathematics items come from SocraTeach, filtered on reference dialogue signals (at least four student–tutor exchanges, at least two annotated student errors, at least one correct or partially correct response after the final error, and no explicit ground-truth disclosure), reducing 35,151 items to 907 questions. Physics and chemistry items come from SciQ, where a keyword-seeded TF-IDF classifier separates domains and retains items classified with confidence ≥ 0.65, sampling up to 500 questions per domain from 11,679 items. The bank therefore anchors conversations but never supplies them.

Each dialogue is generated by pairing one of five Large Language Models (LLMs) tutors with a single fixed simulated student — LLaMA-3.1-8B-Instruct, decoded at temperature 0 — so that within a paired comparison question and student are held constant and any difference is attributable to the tutor's response to the demographic signal. The tutors span model families, scales and training paradigms: LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct (instruction-tuned), TutorRL-7B (pedagogy-tuned with RL), DeepSeek-R1-Distill-Llama-70B (reasoning-tuned), and GPT-5-mini (proprietary frontier). Dialogues alternate turns until the tutor reveals or confirms the correct answer, capped at T_max = 10 turns, against reference dialogues that span 4–7 turns. The experimental conditions vary not only which signal is present but how it is delivered: Explicit (a direct demographic statement), Implicit (a demographically connoted name), and Opposite (a 2 × 2 design in which the student's self-stated gender and the tutor's perceived gender are manipulated independently).

Scoring uses a taxonomy adapted from Maurya et al.'s eight-dimension pedagogical evaluation taxonomy with three changes: mistake_recognize merges identification and location, step_scaffolded and actionability remain separate because pilot annotations showed they vary independently, and Human-likeness and Coherence-with-context are dropped. Four pedagogical functions — Diagnose (mistake_recognize), Reveal (answer_leakage), Next-Step (step_scaffolded, actionability) and Communicate (corrective_tone) — structure the five turn-level metrics, with −1 assigned when a dimension is not applicable. Judge independence is buttressed by four conversation-level metrics (tutor_turns, tutor_avg_word, tutor_question_rate, tutor_student_word_ratio) extracted deterministically from transcripts, which provide triangulation that does not depend on the judge at all.

Evidence: Where Bias Emerges

Bias profiles differ sharply across domains. In mathematics the strongest effects concentrate on proactive behaviors, particularly step_scaffolded (up to |r| = 0.144) and actionability (up to 0.169), while corrective_tone stays uniformly low (0.022–0.073). The pattern reverses in chemistry and physics, where scaffolding and actionability mostly remain below 0.10 but corrective_tone exceeds it for every model (chemistry 0.102–0.168; physics 0.129–0.294). Demographic signals, in other words, primarily affect how a tutor scaffolds mathematics but how it corrects errors in the sciences.

Across demographic dimensions, language and immigration cues dominate. Averaged over the five turn-level metrics, the largest Explicit cells are First Language for DeepSeek-R1-70B (0.281) and GPT-5-mini (0.206) in physics, and Immigration for DeepSeek-R1-70B (0.216). Gender and SES disparities are smaller and more diffuse. Sensitivity to demographic cues is itself a model property: Qwen2.5-7B exceeds the |r| ≥ 0.10 threshold in 7 of 12 domain–dimension cells, whereas LLaMA-3.1-8B exceeds it only once, and DeepSeek-R1-70B pairs the largest language and immigration gaps with the smallest gender bias (0.022 in chemistry). No model exceeds 0.10 on all four demographic dimensions within any domain.

The clearest test of whether capability buys fairness comes from comparing TutorRL-7B with its Qwen2.5-7B backbone. Pedagogy-specific RL suppresses corrective_tone bias (chemistry 0.159 → 0.104; physics 0.294 → 0.163) and answer_leakage bias (chemistry 0.121 → 0.045; physics 0.067 → 0.047), but increases mistake_recognize bias (mathematics 0.102 → 0.165; physics 0.150 → 0.212) and worsens First Language disparities in every domain (mathematics 0.069 → 0.144; chemistry 0.114 → 0.177; physics 0.081 → 0.137). Optimizing for named pedagogical behaviors moves demographic inequality around rather than eliminating it — a finding that bears directly on evaluation practice, since a benchmark scoring only aggregate teaching quality would miss the redistribution entirely.

Ablations: Names, Mismatch, and the Tutor-Side Signal

The Implicit condition isolates whether demographic information must be stated outright. It need not be. Name-mediated pedagogical bias is substantially larger in physics (mean ≈ 0.142) than in mathematics (≈ 0.074) or chemistry (≈ 0.071), and rows combining gender and ethnicity cues ("Both") produce the strongest effects for most tutors, reaching 0.205 for LLaMA-3.1-8B and 0.185 for Qwen2.5-7B. Joint cues therefore generally induce larger disparities than either signal alone, while conversation-level effects stay comparatively small — name-mediated bias primarily reshapes pedagogical behavior rather than dialogue structure. Models also respond differently to cue channel: LLaMA-3.1-8B shows minimal Explicit bias yet reaches pedagogical |r| = 0.205 under Implicit, exceeding all Implicit values of GPT-5-mini, whereas GPT-5-mini is consistently more sensitive to explicit statements.

The Opposite condition asks whether the residual gender signal is tutor-driven or an artifact of the student simulator. Because the student's self-stated gender and the tutor's perceived gender are varied orthogonally, matched (Correct) and mismatched (Wrong) cells decompose the gap into tutor effects, simulator effects and their interaction. Mismatch amplifies disparities in 11 of 15 (tutor, domain) cells: Qwen2.5-7B/mathematics 0.073 → 0.238, TutorRL-7B/mathematics 0.066 → 0.154, DeepSeek-R1-70B/physics 0.043 → 0.169, and GPT-5-mini/physics 0.165 → 0.290. Only four cells reverse the pattern, and all reversals are small (≤ 0.043), spanning all three domains, which suggests model-dependent rather than domain-dependent behavior. Contradictory demographic signals thus amplify divergence beyond what the student persona alone induces, implicating tutor-side demographic processing as a major source of bias.

Implications for Deployment and Fairness Auditing

Bias is measured here with paired non-parametric statistics designed to control for question difficulty: within each (tutor, domain, condition) cell and demographic contrast, every question contributes a matched difference, contrasts require at least 15 paired questions, binary comparisons use two-sided Pratt-corrected Wilcoxon signed-rank tests, multi-level dimensions use Friedman tests followed by Bonferroni-corrected pairwise tests when p < 0.10, and effect sizes are reported as rank-biserial r with 95% percentile bootstrap intervals (B = 1000). Following Funder and Ozer (2019), |r| ≥ 0.10 is treated as practically meaningful and a fairness violation is recorded when the confidence interval excludes zero. GPT-5-mini, evaluated on a fixed stratified 30% subsample per domain (272/150/150 items) for cost reasons, consumed 126,175 API calls and 101,493,943 tokens; the four open-weight tutors used the full bank.

The authors are explicit that EduFair-Bench is an audit instrument, not a certificate. Passing its effect-size thresholds in this configuration does not guarantee equitable treatment of real students, and because all dialogues use one fixed simulated student, bias patterns should be validated against human-subjects studies before strong classroom claims — simulated students are known to diverge from real learners in error patterns, persistence and Help-Seeking. The four demographic dimensions are also coarse, often binary operationalizations that erase within-group heterogeneity, and the Implicit condition covers only gender and ethnicity jointly, so a full intersectional audit across all four axes remains the natural next step. These limitations sit alongside what is nonetheless a hard conclusion for deployment safety: stronger pedagogical capability does not imply greater fairness, and auditing LLM tutors for demographic bias before classroom deployment should be a prerequisite rather than an add-on.

Connected Concepts

• Pedagogical Safety
• Bias Mitigation
• Equity
• Intelligent Tutoring
• Benchmark
• AI Ed Evaluation
• Simulating Students
• Educational NLP
• Scaffolding
• Training Pedagogical LLMs for Tutoring
• Psychometrically Aware AI
• Learner Modeling and Adaptive Instruction
• Human-in-the-Loop

Connected Articles

• SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems — AI Tutor Safety Harms
• SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems — SafeTutors: Pedagogical Safety of LLM Tutors
• LLM Pedagogical Behavior in AI Tutoring Interactions — LLM Pedagogical Behavior in AI Tutoring
• Methodologies for Improving the Quality of AI Tutoring in K-12 Education — AI Tutoring Quality in K-12: Methodologies
• Why Machines Misread Pedagogical Quality: Human-Machine Alignment in LLM-Based Pretest Question Evaluation — Machines Misread Pedagogical Quality
• Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing — Gender Bias Transfer in LLM Writing Support
• Towards Valid Student Simulation with Large Language Models — Valid Student Simulation with LLMs
• ASTRA: A synthetic benchmark for trace-based evaluation of socially intelligent multi-agent tutoring — ASTRA: Multi-Agent Tutoring Benchmark
• AI Tutoring is Not a Monolith: What We Actually Know — AI Tutoring Is Not a Monolith

Citation

Jiaxu Zhao, Bahar Radmehr, Fares Fawzi, Tanya Nazaretsky, and Tanja Käser (2026). EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics. arXiv preprint.