How Do We Train an AI Tutor to Guide Students Rather Than Answer Them?

A general-purpose model is post-trained on human preference for helpfulness, and in practice helpfulness means answering the question promptly and completely. Tutoring requires the opposite: helping a student reach the answer rather than handing it over. This is the one place in educational AI where you may genuinely need to change the model's learned behavior rather than its prompt, and it is also the best-evidenced part of the field. The results are large, and the decisive variable is the reward, not the algorithm.

Why prompting may not be enough here

Prompting and even general alignment training leave a pull toward agreement. Contextual sycophancy persists after prompting and alignment, with learners' errors still propagating into the AI's advice. EduFrameTrap shows that models which withstand context-switch attacks still capitulate under authority or social-affective pressure and withhold corrective feedback — which is why its authors argue that "kind-but-correct" behavior should be an explicit training requirement rather than a preference (Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks).

If your tutor's failure mode is that it agrees with a wrong answer, prompting it not to agree is a mitigation rather than a fix.

The reward is the whole design

A reward is a compressed specification of what you want, and models optimize whatever you actually wrote. That makes reward design the highest-leverage decision in the process, and it is where the best result here was won.

EduQwen's reward model prioritized guiding responses over direct answers, with hard-negative mining to exclude questions the base model already solved, and rollouts extended from 5 to 8 steps so that multi-step pedagogical decisions were captured. Its three-stage pipeline — initial RL, synthetic SFT, final RL — reached 96.52% on the CDPK benchmark, against Gemini-3 Pro's 90.55% (Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised).

Two things follow directly. First, expect the reward to be gamed: it specifies less than you meant, so write it against a benchmark rather than an intuition. Second, the intermediate numbers are instructive — the first RL stage alone reached 94.13%, SFT on 40,000 self-generated responses took it to 96.20%, and the final RL round added the last fraction. Most of the gain came early.

Reinforcement learning beats imitation for pedagogy

If you only do supervised fine-tuning, you teach the model to imitate your demonstrations. That is enough for format and not enough for judgment. LearnLM's finding is explicit: RL is substantially more effective than SFT alone for following nuanced pedagogical instructions in long conversations (LearnLM: Improving Gemini for Learning).

Its instruction-conditioned framing also lets developers and teachers specify tutor behavior without committing to one definition of pedagogy, and it is mixed into Gemini's post-training stages by co-training. Experts preferred it over GPT-4o (+31%), Claude 3.5 Sonnet (+11%) and base Gemini 1.5 Pro (+13%).

Supervise the process, not the answer

The single most actionable finding in this area concerns what the training data must contain. Instruction-tuned models learned algebra misconceptions only when trained on step-level solution traces. Trained on final answers alone, accuracy stayed below 30% at every data size (Misconception Acquisition Dynamics in Large Language Models).

Two further details from that study matter for anyone building either side of a tutor:

• The student role overgeneralized the learned error until correct examples were explicitly mixed in at ratios as low as one in four.
• The tutor role showed no such cost, holding correct accuracy from 93% to 98% across ten jointly trained misconceptions.

If you are training a tutor to diagnose, your examples need the reasoning, not just the verdict. A dataset of question-and-correct-answer pairs cannot teach a model to notice where a student went wrong.

Make the training data carry the pedagogy

If the supervision has to contain reasoning, the labeling scheme is a curriculum decision rather than a data-cleaning step. Two results here bear on that directly.

Label each example with the behavior you want. A study of supervision labels found that assigning every training example one target behavior — subject competence, curriculum grounding, diagnostic reasoning, or scaffolding — lifted every model scale tested, with the largest gains in scaffolding and in using a learner's history. Knowledge-state diagnosis remained the weakest behavior at 54.04%, which is a useful expectation to carry: diagnosis is the hardest thing on this list to teach (OmniEdu: Open Foundation Models for Learning and Teaching).

Treat data selection as a training problem of its own. Edu-QuRating adapts preference distillation to educational data curation, replacing a single "is this educational?" score with 20 rubric dimensions covering factual accuracy, pedagogical structure and level suitability (Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements). If you are assembling a corpus from web text, that is the kind of filter that decides what your model learns to sound like.

Reward density matters as much as the reward

An exact-match reward is too sparse when the thing you care about has several dimensions. SWIM's writing simulator shows the progression cleanly:

• Rubric-grounded prompting: best average trait QWK 0.577 (Claude Sonnet), 0.422 (GPT-5.4), near zero for an open 7B model
• Supervised fine-tuning: 0.474 ± 0.023 for that 7B model
• GRPO with a dense, trait-normalized accuracy reward: 0.618 ± 0.005, across every trait and prompt (SWIM: Student Writing Simulation via Proficiency-Conditioned Generation)

The reward design was the point: a dense trait-normalized signal rather than exact match, because exact match is too sparse in a multi-trait setting. If your reward fires only on a perfect response, most of your training signal is silence.

Train the decision, not only the utterance

Post-training need not only shape what the model says. TACT post-trained a tutor on a 13-strategy taxonomy plus a two-axis student-move taxonomy and gained 20.30 points over its Qwen3.5-4B backbone, with a diagnostic benchmark that withholds the learner-state labels available during training so the model has to infer state from the dialogue (TACT: Taxonomy-Aligned Post-Training for Pedagogically Adaptive English Tutoring).

The same logic runs through special education alignment (Special-R1: Reinforcement Learning for Special Education — Aligning LLM Tutors to Diverse Learners through Disability-Adaptive Training) and through heuristic-RL work that aligns models as Socratic guides rather than answerers (Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning). One platform goes further and trains the policy rather than the prose: a reinforcement-learning agent chooses the next practice problem, so what the learner does next is decided by the trained model (Effective Personalized AI Tutors via LLM-Guided Reinforcement Learning).

What data do you need?

Two opposite bets define the space, and which one you take depends on what you already have:

• Instruction-conditioned post-training when your data is scarce. LearnLM carries system-level instructions that let teachers and developers specify tutor behavior, and relies on co-training rather than a large corpus of tutoring transcripts.
• Fine-tuning on authentic interaction data when you have it. TeachLM bets that prompt engineering is a stopgap and the scarce ingredient is real learner–tutor interaction. Trained on 100,000 hours of one-on-one sessions under rigorous anonymization, it doubles student talk time, improves questioning style, and increases dialogue turns by 50% (TeachLM: Post-Training LLMs for Education Using Authentic Learning Data).

A cheaper route to pedagogical behavior is distillation: Pedagogy-R1 (1.5B and 7B) was instruction-tuned on pedagogically filtered outputs distilled from a QwQ-32B teacher, paired with Chain-of-Pedagogy prompting (Pedagogy-R1: Pedagogical Large Reasoning Model and Well-balanced Educational Benchmark).

Test it at conversation length

Training does not make a model safe over a long conversation. SafeTutors shows that even specialized pedagogical models degrade across sustained dialogue and can commit answer over-disclosure harms (SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems). Pedagogical safety has to be tested at conversation length rather than at the single turn — including the turns where the student is wrong, persistent, or pushing.

Where to go next

• How can we make AI better at supporting learning in our own subject? — whether training is the right lever at all
• How do we know an educational AI is working correctly, not just scoring well? — measuring whether the behavior actually changed
• What are best practices for developing an effective AI tutor? — the interaction-design counterpart, which shapes the same guiding behavior through scaffolding and hint ladders rather than training
• LLM Training and Fine-Tuning — the full concept page