Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

Synthesis: Alignment and evaluation methods for embedding Scaffolding behavior into chatbots rest on an implicit assumption: that students will take up the scaffolding and engage in the conversation.

Key Findings

• The paper introduces an evaluation pipeline around two metrics — Chatbot Scaffolding and Student Uptake — applied across nine datasets of 9,490 chats spanning AI tutor benchmarks and real-world deployments of educational chatbots.
• While benchmarks assume a high-scaffolding, high-student-uptake environment, students in real-world settings exhibit lower levels of uptake overall, frequently bypassing the chatbot's pedagogical framing to drive the interaction toward their own learning goals at little interpersonal cost.
• Bypassing scaffolding is not necessarily detrimental; it frequently highlights a mismatch between a chatbot's pedagogical framing and the student's learning goals.
• Future benchmarks must move beyond the assumption that students will simply take up the scaffolding, and instead evaluate how chatbots navigate diverse learning contexts and student-driven interaction patterns.

Study Design & Method

Scaffolding describes how a tutor calibrates support to the learner's current state — guiding through graduated hints, posing questions rather than giving answers, and withdrawing support as the student gains competence. Delivering timely, dialogic, and scaffolded feedback to every student at every moment of struggle is difficult at scale, and Large Language Models (LLMs)-based chatbots have been proposed as a way to approach this challenge. However, deploying LLMs as tutors introduces a tension: they are trained to be helpful by presenting information and answering directly, rather than engaging students in guided discovery — behavior that is at odds with scaffolding, where a tutor withholds answers to promote reasoning. The evaluation pipeline operationalizes this tension through the Chatbot Scaffolding and Student Uptake metrics, and the corpus spans both benchmark datasets and real-world chatbot deployments.

What this means for practice

• Instructors. Expect students to use an AI tutor as an on-demand learning assistant rather than a conversational partner. Across the real-world deployment datasets, students showed lower uptake overall and frequently bypassed the chatbot's framing, so teach the intended use of a tutoring tool instead of assuming the interface enforces it.
• Instructors. Read a redirecting question as Help-Seeking, not misbehavior. The authors argue that bypassing scaffolding often reflects an instrumental move that exposes a mismatch between the chatbot's framing and the student's goal, and that a highly scaffolded reply is unhelpful to a student who has been stuck for hours and now needs a direct answer.
• Learners. Say what you actually need when a guiding question misses your goal: the analysis found bypassing the chatbot's framing carried little interpersonal cost, and the mismatch it reveals is more useful as a signal about the tool than as evidence of your own disengagement.
• Designers. Decide in advance what the chatbot does when a student does not follow its scaffolding — concede and adapt its behavior, or steer the student back — and let sessions span long gaps, since StemChat, StudyChat, and MathsChat show students leaving and returning across topics instead of holding one continuous conversation. Build scaffolding to adapt to student-driven interaction patterns, including Help-Seeking styles, rather than presupposing the conversational sequence the interface expects.
• Researchers. Score the student's side of the exchange, not only the chatbot's turn: the pipeline's Chatbot Scaffolding and Student Uptake metrics ran across nine datasets of 9,490 chats, and uptake was only inconsistently linked to the degree of scaffolding a chatbot provided. A benchmark that assumes uptake will overestimate both the value of rigid scaffolding and the quality of the LLM tutor behind it.

Limitations

• Both metrics are scored in a single LLM-judge prompt call using GPT-5-mini, chosen on cost grounds; the authors acknowledge the single call may influence the output score and leave a split-prompt ablation to future work.
• The authors describe metric validation as only an initial investigation: two human raters reviewed 282 chats stratified by score, agreement with the judge was moderate, and most disagreements fell on chats binned at 0.5 in the benchmark datasets.
• Face-to-face human tutoring was deliberately excluded, so the analysis covers only chat-based student–tutor exchanges and cannot address the visual cues, intonation, or gesture that in-person tutoring carries.
• The scaffolding rubric is derived from the AI tutor benchmarks and defines scaffolding over the trajectory from impasse to a correct solution, so turns that acknowledge a student's correct answer and restate it are scored as low scaffolding even though they appear regularly in real-world deployments.

Citation

Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer, Rhodri Nelson, Peter B. Johnson (2026). Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments. Pluralistic Alignment Workshop @ ICML 2026.