From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

Synthesis: This paper tackles a core ITS challenge: predicting when students will disengage so tutors can intervene before it's too late. It introduces engagement forecasting as a supervised prediction task with two complementary targets: minutes practiced per week (effort) and new skills mastered per week (progress).

Overview

Benchmarking 15 predictors on 425 middle-school students:

• Feature-based models (regressions, decision trees, neural networks) reduce MAE by 22–33% compared to heuristic baselines
• Percentile heuristics (adapted from prior behavioral domains) systematically overpredict — they assume past behavior continues, missing disengagement signals
• Feature-based models better track actual practice trajectories week-to-week

Distinct predictive signatures for effort vs. progress:

• Effort forecasting driven primarily by recent activity features (how much the student practiced recently)
• Progress forecasting depends more on learner-state and content difficulty signals (what the student knows and how hard the material is)

Human validation: Semi-structured interviews with 8 college tutors confirmed that tutors reason differently about effort goals vs. progress goals, mirroring the model's feature importance patterns. This strengthens the case for practical deployment.

Connections to the ITS Research Landscape

This paper extends the Comprehensive Review of Intelligent Tutoring Systems findings on what makes ITS effective by adding a temporal prediction layer. Where prior work evaluates whether tutoring works on average, engagement forecasting asks when it works and for whom — connecting to the personalized intervention paradigm in ProPACT: A Proactive AI-Driven Adaptive Collaborative Tutor for Pair Programming.

The focus on middle-school students (N=425) aligns with the The Evidence Base on AI in K-12: A 2026 Review, which calls for more rigorous K-12 efficacy studies. The EDM 2026 venue, combined with Not All Students Engage Alike: Multi-Institution Patterns in GenAI Tutor Use, suggests engagement analytics is becoming a recognized subfield within educational data mining.

Methodological Contribution

The paper establishes a reproducible Benchmark for engagement forecasting, with clearly defined prediction targets, a documented feature set, and public interaction log data. This is significant for the benchmark landscape in AIED, where many systems are evaluated on proprietary data with incomparable metrics.

What this means for practice

• Designers. Forecast minutes practiced and new skills mastered as two separate week-ahead targets, since effort is driven by recent-activity features while progress depends on learner state and content difficulty.
• Instructors. Use the forecast to set a weekly target with the student rather than flagging disengagement only after it has happened, because percentile heuristics systematically overpredict and assume past behavior continues.
• Designers. Replace percentile-style heuristics with feature-based models; on the 425-student benchmark they cut mean absolute error by 22–33% relative to the heuristic baselines.
• Researchers. Re-validate any deployment at a different site: the models were fit within a single ITS, subject domain and logging schema, and the authors call for cross-site replication and leave-site-out testing.

Limitations

• The study is confined to a single ITS, subject domain and platform-specific logging, and features such as opportunity counters, fine-grained timestamps and skill tags for AFM are not universal and may need proxies or simplified learner-state estimates elsewhere.
• The analysis does not distinguish classes that assign homework from those that do not, though the authors note these contexts differ in how student agency and self-regulated learning operate.
• Both prediction targets are imperfect proxies for effort and learning progress, and alternative proxies such as problem attempts, session frequency and active-versus-idle time were not compared in this work.
• Human validation came from interviews with only 8 college tutors, while the forecasting evaluation used 425 middle-school students, so the tutor reasoning evidence rests on a different population and a much smaller sample than the predictive benchmark.

Citation

Qiu, E. S., Thomas, D. R., Guo, B., Aleven, V., & Borchers, C. (2026). From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning. EDM 2026.