TutorCaseBench: A Case-Grounded Benchmark for Evaluating LLM Tutors in Supporting Students with Challenging Behaviors

Synthesis: Yeon Su Park and colleagues (2026) ground benchmarking of AI tutors in a source the field had left unused: qualitative case studies from pedagogy, educational psychology, and Special Education, where practitioners document individual students, the events that set off refusal or withdrawal, and which responses were observed to work. An automated pipeline converted 4,755 ERIC candidates into 923 six-turn tutor–student dialogues from 300 papers and 518 student categories, sorted into five challenge types and scored against case-specific rubrics instead of fixed pedagogical dimensions. Across 11 models, the best — Claude Opus 4.7 — satisfied a documented intervention in 55.8% of open-ended cases and 43.3% of long-term ones, while the same model chose the right strategy from four options 63.0% of the time. That gap between recognizing an intervention and producing one is the paper's central result, and it bears directly on whether Conversational AI tutors can hold a disengaged or distressed learner in a real classroom. All dialogues, rubrics, and items are public at https://github.com/kixlab/TutorCaseBench.

Key Findings

1. The best model still misses most cases. Claude Opus 4.7 satisfied at least one rubric criterion in 55.8% of the 1,560 open-ended items, and in only 43.3% of the 736 long-term intervention items.
2. Selecting beats generating. On 2,537 four-option items the best model reached 63.0%, and even the lowest, GPT-OSS 120B, reached 49.2% against a 25% chance level.
3. Every model is worse at sustained plans. All 11 did better on immediate responses; Gemini 3.1 Pro fell from 67.5% to 35.2% and GPT-5.4 Mini from 49.2% to 33.3%.
4. Open-weight models trail. DeepSeek V4 Pro led open-weight models at 47.0%; Llama 4 Maverick was lowest at 31.7%, and its most frequent 4-gram, "take a deep breath," appeared in 50.4% of its immediate responses.
5. Social conflict is the hardest category. Social-interpersonal difficulty ranked last for 6 of 11 models on immediate open-ended responses and was the weakest immediate category for all 11 in multiple choice.
6. Failures cluster into four moves. In 50 cases all models failed: empathy or praise where the case prescribed confronting a belief, generic scaffolding, over-intervention, and missed task redesign.
7. References help the next turn, not the plan. Similar-case strategies lifted 10 of 11 models by 9.9 points on average immediately (Llama 4 Maverick, 41.0% to 64.8%) but cut long-term performance by 5.2 points overall.

What the corpus is built from

The pipeline starts at ERIC, restricted to 2010–2026, with papers that carry at least one student descriptor and one methodology descriptor; 4,755 candidates survived into screening, where each was rated 1–5 on how concretely it documented student behavior, learning context, and teacher actions, and papers scoring below 3 on any dimension were dropped. Two extraction passes then pulled verbatim learner evidence — observable behavior, cognitive state, affective state, and difficulty triggers — and instructional evidence tagged effective or ineffective with an immediate or long-term scope. That yielded 923 scenarios from 300 papers and 518 categories (M = 1.78 scenarios per category). Task resistance is the largest slice at 28.7%, followed by attention/engagement failure at 20.4%, social-interpersonal difficulty at 18.9%, academic struggle at 18.4%, and emotional distress at 13.7%. Rubrics hold 2.90 immediate and 2.45 long-term criteria on average, and a response succeeds if it satisfies any one of them.

Recognition is not generation

The multiple-choice task pairs one documented effective strategy with three from other scenarios, and its scores run higher and tighter than the open-ended task: 49.2% to 63.0%, against 31.7% to 55.8%. A binary variant that contrasts an effective with an ineffective strategy from the same case produced 85.8% to 93.1% accuracy, far above the 50% baseline — so the weaker four-option result is not an inability to reject bad advice. Models can tell a documented intervention from an ineffective one; what they cannot reliably do is choose among several plausible, individually defensible interventions for the student in front of them. The authors read this as the difference between intervention recognition and context-specific adaptation. For evaluation of AI systems, that distinction matters because a strategy-selection score would make tutors look nearly ready, while the free-response score, which is closer to what a student actually receives, does not.

Where the models default

Qualitative analysis of 50 cases that every model failed names four recurring moves. In one, a student who believes her work is wrong calls herself "stupid"; the documented response is to verify the work before she presents, while models reassure her that mistakes are acceptable. In another, a student reads "5 times more" additively; the case prescribes simpler terms such as "double" or "triple," while models correct the arithmetic and leave the language difficulty untouched. A third shows over-intervention: a quiet student co-teaching a science lesson should be left to lead, but models keep prompting him to breathe and speak louder. The fourth misses task redesign entirely, calming a student and returning him to a reading task the case says should be replaced. Errors also converge: all 11 models answered 26.8% of multiple-choice items correctly and missed 17.5%, and among the items all of them missed, all 11 picked the same distractor in 40.3% of cases — a pattern of shared, not idiosyncratic, blind spots in Scaffolding and social-emotional support.

Why references move one turn at a time

Adding strategies from similar prior cases — three retrieved from a pool of 255 papers and 785 scenarios, never from the test dialogue's own source paper — raised immediate-response success for 10 of 11 models by 9.9 points on average, with Llama 4 Maverick climbing from 41.0% to 64.8% and Gemini 3.1 Pro unchanged at 71.3%. The same intervention hurt long-term planning: 7 of 11 declined, and the average fell 5.2 points. References work when they can guide the next tutor move; a plan spanning weeks cannot be assembled from three borrowed strategies. Two correlational checks point the same way: lexical diversity tracked immediate success (ρ = 0.800 for distinct 4-grams, p < 0.01), and case-recovery rate — whether a response sits closer to other responses to the same case than to responses to different cases — correlated at ρ = 0.691. Stronger models are less templated and more case-specific, which is what skilled teaching depends on and what a fixed rubric would not have detected.

What this means for practice

• Instructors. Do not hand the behavioral moment to a chat tutor unassisted: supply the trigger and what has worked with this student before, or take the turn yourself, because the strongest model met a documented response in 55.8% of cases.
• Faculty developers. Teach from documented cases rather than strategy inventories. Models picked a documented strategy 63.0% of the time but produced one far less often, and the same gap appears in teachers who know the label "scaffold" without seeing this student's need.
• Developers. Retrieve similar cases for the next turn, not for multi-week plans: immediate success rose 9.9 points on average with references, while long-term performance fell 5.2.
• Administrators. Ask vendors for evidence on challenging-behavior cases, not only on math-error detection. Social-interpersonal difficulty was the weakest immediate category for all 11 models, and the failures converged on the same wrong answers.
• Researchers. Build benchmarks from documented cases rather than simulated personas, then cluster categories across the corpus — the authors' own next step — to test whether these failure modes recur.

Limitations

• Construction ran almost entirely on one model family: gemini-3-flash-preview for extraction and gemini-3.1-pro-preview for generation. Human validation of 30 papers by 12 reviewers rated 96.6% of 882 judgments Yes or Partly, but exact agreement between the two reviewers per paper was 58.2%, and only 43.4% for instructional strategies.
• The corpus is geographically narrow: 61.4% of the 518 categories come from North America, 58.9% from the United States, 0.4% from Africa, and 41.9% are primary-school K-12 students against 5.4% tertiary, so university tutoring is thinly represented.
• Evaluation is text-only and turn-bounded. Dialogues describe nonverbal cues but cannot test whether a tutor reads a room, behavior detection is explicitly out of scope, and four-, six-, and eight-turn variants scored 45.8%, 46.6%, and 46.8% — the difficulty is not dialogue length.
• Success is defined as satisfying at least one of 2.90 immediate or 2.45 long-term criteria, so 55.8% is a lenient bar rather than evidence of full-fidelity support.
• All 11 tutors are 2025–2026 API releases (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro, Llama 4 Maverick) evaluated in 2026 against case literature from 2010–2026, and categories were built within papers rather than clustered, which blocks analysis of recurring patterns.

Citation

Park, Y. S., Park, C., Kim, T. S., Kim, S., Kim, H., Jung, D., Yoon, J., & Kim, J. (2026). TutorCaseBench: A Case-Grounded Benchmark for Evaluating LLM Tutors in Supporting Students with Challenging Behaviors. arXiv preprint arXiv:2610.30828.