Incremental Predictive Value of LLM Tutoring Interactions for Student Performance

Knowing that students talked to an LLM tutor a lot says surprisingly little about how they did. This study turns 16,851 student–assistant exchanges from two offerings of an undergraduate AI course into semester-level interaction features and asks whether those features beat a constant baseline in predicting assignment scores. They barely do: the best model (Gradient Boosting) reduces student-level mean absolute error from 3.138 to 3.023 grade points, a difference whose 95% interval crosses zero. The paper's contribution is methodological, not predictive. Because 160 of 175 matched students scored above 90, a ceiling leaves almost nothing to explain, and error is concentrated in a small low-scoring stratum. The authors argue that learning analytics studies must report against baselines and name the students who carry the model's error, not just quote accuracy.

Key Findings

• The predictive gain is small and not distinguishable from zero. Gradient Boosting's student-level MAE is 3.023 against 3.138 for the median baseline, a 0.114-point difference with a conditional 95% interval of [−0.245, +0.013].
• A ceiling effect undercuts the whole task. Of 175 matched student–semester records, 160 (91.4%) scored above 90, so the outcome variable has little room to vary and constant baselines do well by default.
• Error sits in a small stratum. The lowest scoring quartile holds 25.7% of the matched records but contributes 62.1% of Gradient Boosting's absolute error, so aggregate MAE hides who the model fails.
• Linear models lose to a constant. Ridge (3.331) and Elastic Net (3.344) are worse than the median baseline, and all six fitted models have negative mean fold R².
• Feature ablations move the result by a few hundredths of a point. Dropping request-type ratios gives 3.070 and dropping prompt, conversation-count and active-day features gives 3.079, while all 15 features give 3.024.

Why the study asks for a baseline, not a score

Predictive Learning Analytics papers often report model accuracy without a reference point, which makes a modest model look informative and a useful one look mediocre. The design here fixes that: six supervised estimators share repeated nested cross-validation with two constant baselines (mean and median) across 50 folds and ten repeats, and differences are reported as paired intervals from 5,000 draws. Under that framing, the honest summary of a tutoring-interaction model in this cohort is that it is roughly level with guessing the median.

The representation is deliberately semester-level rather than turn-by-turn: volume of exchanges, request-type ratios, conversation and prompt counts, active days and follow-up ratio, fifteen features in all, computed for each student–semester. The question is the one an advisor would ask of a dashboard, whether a term's worth of Help-Seeking behavior carries signal about assessed performance, and the answer turns out to depend heavily on how much variance the Assessment itself leaves.

What the ceiling and the quartile reveal

Two properties of the data do most of the explanatory work. The first is the ceiling: with 160 of 175 students above 90, any model is being asked to predict a variable that is nearly constant, so even a genuinely informative feature set would show a tiny increment and a baseline that always predicts the median would already be close. The second is where error lives. Boosting's mistakes are not spread evenly but concentrated in the lowest quartile, which means a model that looks near-equivalent to a baseline on average can be systematically wrong for exactly the students a support system would target.

The ablation results then read as a warning about feature stories. Removing 15 features down to almost none costs about 0.05 grade points, and dropping the follow-up ratio costs nothing measurable. Feature importance narratives built on a dataset shaped like this would describe a model that is mostly fitting noise in a compressed outcome range.

Reading the result as a design constraint

The authors are explicit that the study does not demonstrate a learning effect: semester-level interaction volume is a proxy, not a mechanism, and the cohort is two offerings of one course. What it does demonstrate is a reporting standard. A tutoring platform that wants to claim its interaction data predicts outcomes should publish the constant baseline beside every model, report per-stratum error so the affected students are visible, and test prospective prediction across courses rather than within one. For chatbot deployments the practical implication is that engagement logs are not a performance measure; they are an input whose value has to be established against an alternative that predicts nothing at all.

What this means for practice

• Compare every tutoring-analytics model to a constant baseline. Report the median-prediction MAE beside the model's, as this study does, before telling instructors the data has predictive value.
• Break error down by score band, not just whole-cohort averages. A model level with the baseline overall can be badly wrong for the lowest quartile, which is where support decisions are actually made.
• Treat interaction volume as a proxy until a cross-course test exists. A term's chat count reflects course design and task affordances as much as learning, so use it for formative conversation, not for prediction claims.
• For learning-analytics designers, prefer this evaluation harness. Repeated nested cross-validation with paired intervals and explicit error populations gives a defensible verdict on whether a feature set earns its place in a dashboard.

Limitations

• The cohort is 175 matched student–semester records from two offerings of a single undergraduate AI course, analyzed after the fact from a released dataset rather than collected for prediction.
• Assignment scores cluster above 90 (160 of 175), so the ceiling constrains the achievable effect and the observed gains cannot be read as an upper bound on predictive value elsewhere.
• All models have negative mean fold R², meaning none explains variance better than a constant across folds; the point estimates are therefore comparisons among weak alternatives.
• Interaction features are semester aggregates that cannot separate a student who asked many shallow questions from one who asked few deep ones.
• No prospective or cross-course validation is reported, and no intervention tests whether acting on these predictions improves anything.

Citation

Dong, Z., Kuada, M. K., Holley Buchakjian-Tweedy, A., & Aljabea, I. (2026). Incremental Predictive Value of LLM Tutoring Interactions for Student Performance. arXiv:2610.33488.