TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education

Synthesis: Barron et al. (2026) present TutorTrace, a dataset and behavioral-abstraction pipeline that makes learners' behavioral context visible and computable in real time from low-level IDE telemetry during AI-assisted programming education. Across four deployments in two introductory Python courses (N=480), the system captures ~180K telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics, from which it derives a taxonomy of learner activity before the first AI query, between consecutive queries, and across the full session. In a preliminary classroom evaluation, behavior-aware prompts reduced intervals between queries with no independent work from 50.0% to 20.7%, and the data support held-out prediction of imminent queries (AUROC=.726) and guided versus dependent Help-Seeking (AUROC=.717). The work shows how behavioral context can enable adaptive AI tutoring at scale.

From learner queries to learner behavior

AI programming tutors provide scalable support but lack the behavioral context human tutors rely on to adapt support to learners' needs. Human tutors respond not only to what learners explicitly say but to what their observable behaviors suggest they implicitly need — whether a student is struggling or has simply not put in the effort. Current AI tutoring systems are typically limited to the question itself, missing the struggle (or lack thereof) that preceded it. TutorTrace addresses this gap by capturing behavioral context from low-level IDE telemetry and making it computable in real time, enabling systems to respond not just to what learners say, but to what they have done leading up to the Help-Seeking moment.

Dataset and behavioral-abstraction pipeline

TutorTrace was deployed across four course offerings in two introductory Python courses (N=480 learners). It captures approximately 180,000 telemetry events, 13,633 behavioral segments, and 27 continuously computed metrics. The pipeline abstracts raw IDE events into meaningful behavioral states — distinguishing, for example, periods of independent work from periods of inactivity before an AI query. From this foundation, the authors derive a taxonomy of learner activity in three temporal windows:

• Before the first AI query — establishing the learner's baseline engagement and independent-work behavior.
• Between consecutive queries — capturing whether the learner worked independently in the gap between help-seeking moments or simply waited.
• Across the full session — characterizing overall patterns of engagement and reliance.

This taxonomy is designed to feed downstream tutoring systems, giving them behavioral context comparable to what a human tutor observes.

Empirical evaluation

In a preliminary classroom evaluation, behavior-aware prompts — which use the learner's recent behavioral state to shape how help is offered — were associated with a decrease in intervals between queries with no independent work from 50.0% to 20.7%. This is direct evidence that making behavioral context visible changes learner–AI interaction in productive ways. As an additional demonstration of downstream utility, the authors evaluate TutorTrace on two held-out prediction tasks:

1. Whether a learner will query within the next 60 seconds (AUROC = .726).
2. Whether an upcoming query reflects guided or dependent help-seeking (AUROC = .717).

These results connect TutorTrace to Learning Analytics, Learner Modeling and Adaptive Instruction, and real-time intervention in computing education, and demonstrate that behavioral telemetry can predict and classify help-seeking quality.

Connection to the broader knowledge base

TutorTrace speaks directly to the knowledge base's research on Help-Seeking and its failure modes. Prior work documents premature hint requests, superficial hint reading, and answer-seeking over learning-seeking as unproductive behaviors in intelligent tutoring systems. TutorTrace offers an automated, real-time way to detect the behavioral precursors of these patterns — for example, distinguishing a learner who works independently before querying (guided help-seeking) from one who queries with no prior independent work (dependent help-seeking). This supports the design principle of Scaffolding how students ask rather than merely providing answers, and aligns with Student-AI Interaction research on the quality of learner–AI dialogue. The behavior-aware prompting result also shows that AI-driven adaptation need not erode engagement; rather, well-timed interventions can push learners back toward productive, independent effort.

What this means for practice

• Designers. Condition help on observed behavior rather than on the query alone: behavior-aware prompts that used the learner's recent behavioral state reduced intervals between queries with no independent work from 50.0% to 20.7%.
• Designers. Use the three taxonomy windows — before the first query, between consecutive queries, and across the session — as the feature set for detecting passive re-querying in real time instead of relying on question text.
• Designers. Treat the classifier outputs as approximations when wiring them into intervention logic: imminent-query prediction reached AUROC = .726 and guided versus dependent Help-Seeking reached AUROC = .717, both from one tutor model and a rule-based segmentation pipeline.
• Researchers. Validate behavior-aware interventions under randomized assignment before reporting effects, because the preliminary classroom evaluation contrasted a morning baseline deployment with an afternoon intervention deployment.

Limitations

• Data were collected during short introductory programming tasks at a single institution (480 learners across four deployments in two introductory Python courses), which the authors state limits generalizability across courses, tasks, populations, and learning environments.
• The classroom evaluation was not randomized: a morning baseline deployment was compared with an afternoon intervention deployment, so cohort composition, time of day, or other unmeasured factors may explain the drop from 50.0% to 20.7%.
• Guided and dependent help-seeking labels were generated by GPT-4o and validated against two human raters on only a subset of queries, and the whole system uses that single tutor model, so the labels are approximations rather than direct measurements of cognitive engagement.
• The rule-based auto-segmentation pipeline lacks semantic understanding of learner code and intent — it can label edits as debugging while an unresolved error remains — and the Window 1 and Window 2 profiles were selected through an exploratory procedure partly informed by task-completion differences, making their completion rates descriptive rather than independently validated.

Connected Concepts

• CS Education
• Intelligent Tutoring
• Help-Seeking
• Learning Analytics
• Student-AI Interaction
• Learner Modeling and Adaptive Instruction
• Student Engagement
• Agentic AI
• Scaffolding
• Large Language Models (LLMs)
• Generative AI
• Student Experience

Connected Articles

• Efficiency vs. Effectiveness: Self-Regulated Learning with LLM-Mediated Help-Seeking — LLM-mediated help-seeking in STEM: layered, instrumental, and verified
• Revisiting the Hint Button: Consistent Negative Associations Between Unproductive Hint Use and Learning Outcomes in Intelligent Tutoring Systems — Premature hint requests and superficial hint reading predict lower learning gains in an ITS
• Beyond Access: Guided LLM Scaffolding for Independent Learning in Undergraduate Statistics — Guided LLM scaffolding improves reasoning-focused help-seeking
• Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments — The scaffolding/student-uptake mismatch in real-world LLM tutor deployments
• The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course — The StudyChat dataset of student–LLM dialogues in an AI course
• One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment — One Click Away: Khanmigo in a two-year school experiment
• Virtual Tutoring with Computer-Assisted Learning: An Experiment in Take-Up and Learning — Virtual tutoring with CAL: take-up and learning
• Regulating the AI Tutor: Intentions, Help-Seeking, and Self-Regulated Learning in Adolescent GenAI Use — The intention-behavior gap in adolescent GenAI help-seeking
• Warning About AI Fallibility Increases Help-Seeking in an Intelligent Tutoring System — Warning about AI fallibility increases help-seeking
• Analysis of Types of Inquiries in Student-AI Interaction: A case study of two CS2 tasks — Analysis of types of inquiries in student-AI interaction
• Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming — GenAI performance on authentic introductory OOP assessments
• Trust and Reliance on AI in Education: AI Literacy and Need for Cognition as Moderators — Trust vs. appropriate reliance during Python problem-solving

Citation

Barron, D., Tang, X., Dwisantika, R., Kim, M., Smith, D. H., Cui, J., & Chen, Y. (2026). TutorTrace: A dataset and taxonomy for classifying learner behavioral states during AI-assisted programming education. arXiv:2608.26184 / UIST '26.