MUSE: Revealing Responsiveness Gaps of LLM Tutors on Multiple-Solution Programming Problems

Synthesis: Cen and Sonkar (2026) introduce MUSE, a student-simulation environment that fixes three things at once — the student's chosen solution, whether the student's intuition about it is correct, and whether the tutor opens with a probe or a direct hint — so that LLM tutors can be observed on programming problems that admit more than one correct answer. Across 150 NeetCode problems and four open-weight tutors, tutors rarely committed to the optimal approach: brute force took 22.2 to 32.1 percent of solution-specific openings while the optimal solution took 13.0 to 19.4 percent, and tutors frequently redirected students away from a valid choice toward their own. The failure is symmetric. Once a student finished, tutors usually stayed anchored when a stronger approach was available, and in correct-intuition probe-first conversations the four base models moved students toward a worse approach in 6.1 to 7.1 percent of cases. Supervised fine-tuning on synthetic dialogues that demonstrate responsive questioning raised the Solution Support Rate by 9.9 and 17.7 percentage points for correct and false intuitions, so responsiveness appears to be trainable rather than fixed in the base model.

Key Findings

1. Tutors carry a solution preference. Among solution-specific openings, brute force was the most preferred approach in all four models (22.2 to 32.1 percent), while the optimal approach took only 13.0 to 19.4 percent.
2. Some tutors avoid committing to any solution. Llama gave general advice in 53.9 percent of openings against 30.0 percent for GPT-OSS, so models differ in whether they name a solution at all.
3. Probing first helps, and unevenly. Under probe-first openings, GPT-OSS, Qwen and Gemma raised Solution Support Rate by 19.8, 11.7 and 12.7 percentage points for correct intuitions, with weaker effects for false intuitions.
4. Supporting a student and extending a student are separate skills. Qwen combined relatively high responsiveness with low post-solution transition scores, showing that accepting a student's approach does not imply introducing a better one afterward.
5. Tutors sometimes move students backwards. In correct-intuition probe-first conversations, all four base models guided students toward a less optimal approach in 6.1 to 7.1 percent of cases, undoing progress already made.
6. Fine-tuning buys responsiveness. Gemma SFT1 raised Solution Support Rate from 0.507 to 0.606 for correct intuitions and from 0.419 to 0.596 for false ones, gains of 9.9 and 17.7 points over the base model.
7. Solution context buys better transitions. Gemma SFT3, trained and evaluated with the candidate solution space in the prompt, lifted the Post-Solution Transition Score to 0.936 and 0.903 and eliminated less-optimal transitions for correct intuitions.

How a simulated tutorial is held constant

Each problem carries three classified solutions — brute force, suboptimal and optimal — ordered by time complexity, and the simulated student pursues one of them at each turn while a flag records whether the intuition attached to it is correct or flawed. The tutor's opening is either direct (solution-specific guidance or general advice) or probe-first, and a separate LLM judge sorts every tutor turn into brute force, suboptimal, optimal or general advice. Two metrics follow. The Solution Support Rate counts a tutor as responsive when it guides toward the student's own approach or offers general advice, which applies to every solution. The Post-Solution Transition Score weights the move made after the student completes a solution: plus one for stepping up to a more efficient approach, minus one for stepping down, zero for staying. Because the student's side is fixed by the environment rather than generated by a live learner, the only thing varying between conditions is the tutor, which is what allows preferences and transitions to be compared across models on the same problems — a reusable Benchmark rather than a one-off evaluation.

What the preference data show

The two opening conditions answer different questions. Under direct openings, the tutor reveals a preference before seeing any student work, and that distribution is where brute force dominates: 22.2 to 32.1 percent of solution-specific responses against 13.0 to 19.4 percent for the optimal solution, with general advice absorbing the remainder. The authors offer two explanations without choosing between them — brute force is simpler to explain in stages, and it is likely more prevalent in the code-heavy text these models were trained on. Under probe-first openings, the tutor asks what the student has tried — the leading question rather than the answer — which reduces the chance that it has already committed to its own approach. That is why responsiveness improves more for correct intuitions than for false ones: when the student's reasoning is sound, a tutor that has not yet named a solution has little to redirect away from, and when it is unsound, general advice alone does not repair it.

What the transition data show

Post-solution behavior is worse than the opening behavior in an instructive way. All base models show positive transition scores, so there is a general pull toward more efficient solutions, but the magnitude varies widely: Llama moves most readily toward optimization and Qwen stays most anchored to the student's own method. Each of the four base models, though, also takes the student backwards in a measurable fraction of correct-intuition probe-first conversations. The fine-tuning results then split the problem into two parts. SFT1, trained without any solution list, raises responsiveness on its own, so the behavior of building on a student's reasoning is learnable from examples. SFT3, given the candidate solutions at both training and inference, barely differs from SFT1 on responsiveness but pushes transition scores much higher. The authors' reading is that fine-tuning teaches how to make a transition while the solution list tells the model which alternatives exist — the skill and the knowledge arrive through different channels.

What this means for practice

• Instructors. Do not assume an AI tutor is neutral among valid approaches. Before using one in a course, ask it the same problem twice with different starting strategies and check whether it follows the student or redirects to its own preference.
• Developers. Log two numbers, not one: how often the tutor builds on the student's current approach, and what direction it moves once the student finishes. The paper shows these come apart, with high responsiveness coexisting with low post-solution transition scores.
• Developers. Supply the solution set explicitly when transitions matter and train responsive behavior separately. The ablation indicates that listing alternatives does not teach responsiveness, and that responsiveness training without the list does not produce reliable improvement moves.
• Researchers. Use a scripted student to hold the learner side constant when comparing tutors; field data cannot separate a tutor's preference from the students it happened to meet.
• Curriculum designers. Treat multiple valid solutions as an asset for learning to solve problems with a failure mode. A tutor that steers every student to the optimal path removes the comparison between approaches that makes the problem worth assigning.

Limitations

• The four tutors were GPT-OSS 120B, Qwen3.8-27B, Llama 3.3 70B Instruct (served as FP8) and Gemma 4 12B IT, with GPT-OSS 120B also acting as the judge for every conversation, so one model family supplies both a tutor and the scoring.
• In this programming education setting the item bank is 150 curated interview-style NeetCode problems with three solutions apiece, which the authors say may not capture the diversity of multiple-solution problems; they propose extending MUSE to mathematical proofs and physics.
• Students are simulated, not real: their correct and false intuitions were generated by GPT-OSS 120B, so the conversations test responses to plausible scripts rather than to actual learners.
• The fine-tuning study covers a narrow configuration — LoRA on Gemma 4 12B IT with rank 32, alpha 64, three epochs, about 640 synthetic conversations from 120 training problems, and three random splits of 30 test problems.

Citation

Cen, S., & Sonkar, S. (2026). MUSE: Revealing Responsiveness Gaps of LLM Tutors on Multiple-Solution Programming Problems. arXiv preprint.