Architecting an AI-Driven Decision Support System for Enhanced Online Learning and Assessment

Synthesis: This technical review synthesizes 2020–2025 research on AI-based decision support systems (AI-DSSs) for online learning and assessment, integrating machine learning, NLP, knowledge-based systems, and deep learning to enable predictive analytics, automated grading, and personalized learning paths. The authors propose a modular four-component architecture (data collection, AI processing, decision engine, user interface) integrated with LMSs via LTI, and report concrete gains including up to 70% faster grading and 12–20% grade increases. It critically assesses technical, ethical, and implementation barriers—interpretability, bias, privacy, cost, and adoption resistance—alongside mitigation strategies and future directions toward generative AI and Multimodal AI integration.

Key Findings

• A proposed modular AI-DSS architecture (data collection, AI processing, decision engine, user interface) integrates with existing LMSs via Learning Tools Interoperability (LTI), featuring federated learning for Privacy and a hybrid RL–ML decision engine for dynamic path optimization.
• Concrete performance gains reported across the literature: automated NLP essay grading achieves up to 0.85–0.90 correlation with human graders and reduces grading time by up to 70%; adaptive reinforcement learning systems improve student retention and performance by roughly 12–20%.
• AI techniques show complementary strengths: supervised ML predicts outcomes with 85–90% accuracy, deep learning reaches up to 92% precision on multimodal assessments, and NLP chatbots answer queries with ~90% accuracy.
• Major limitations persist: LLMs hallucinate in 15–25% of responses, reinforcement learning suffers sample inefficiency and reward-design instability (20–30% outcome variability), and deep learning incurs high computational costs (raising costs 40–50% in resource-limited settings).
• Evaluation relies on a balanced mix of quantitative metrics (accuracy, precision, recall, response time, scalability) and qualitative frameworks (user satisfaction, engagement), with benchmarking against non-AI LMS baselines showing NLP grading up to 80% faster and ML recommendations improving retention 15–20%.

Study Design & Method

This is a technical review (not a meta-analysis) following PRISMA guidelines for rigor and transparency. A systematic literature search covered IEEE Xplore, Scopus, and Web of Science using keywords including "AI-based decision support," "online learning," "automated assessment," and "adaptive learning." The search targeted peer-reviewed journal articles, conference papers, and technical reports published between January 2020 and July 2025. Inclusion prioritized studies with empirical evaluations, novel AI-DSS frameworks, or real-world implementations; exclusion removed non-peer-reviewed sources and studies lacking technical depth. The review synthesizes findings through case studies of prominent platforms (e.g., a MOOC using NLP, an adaptive learning system using reinforcement learning) and comparative analyses of ~29 summarized studies (Tables 5–6), and proposes an implementation/validation roadmap (prototype → controlled pilot → A/B trials → multi-institution deployment).

What this means for practice

• Designers. Build the four components deliberately — data collection, AI processing, decision engine, user interface — and integrate through LTI into the LMS teachers already use, since the review specifies a modular layer rather than a replacement platform.
• Designers. Plan the validation ladder before deployment: the review's roadmap runs prototype, controlled pilot, A/B trials, then multi-institution deployment, which is the sequence that establishes whether a decision engine actually changes learner outcomes.
• Administrators. Budget for the documented failure modes rather than treating them as future work — hallucination in 15-25% of LLM responses, 20-30% outcome variability from reward design, and 40-50% higher costs in resource-limited settings — and require human oversight, model interpretability, GDPR and FERPA compliance, and regular fairness audits alongside deployment.
• Administrators. Make accessibility, multilingual support, and bias-aware algorithms procurement criteria, because the review's user-centric design requirement is where equity is decided rather than in the model choice.

Limitations

• The paper is a technical review, not a meta-analysis: it reports performance figures from 29 summarized studies across two summary tables rather than pooling or re-analyzing them, so numbers such as "up to 70%" faster grading or a 0.85-0.90 correlation with human graders cannot be compared across systems.
• The evidence window is January 2020 to July 2025 across IEEE Xplore, Scopus, and Web of Science, admitting conference papers and technical reports while excluding non-peer-reviewed sources, which bounds what the synthesis can say about deployment outside those venues.
• The proposed four-component architecture is not itself evaluated: the modular design and LTI integration are a proposal, and validation is set out as a roadmap (prototype, controlled pilot, A/B trials, multi-institution deployment).
• The field-level weaknesses it catalogs are the primary studies' own reported results — 15-25% LLM hallucination, 20-30% outcome variability under reinforcement learning, 40-50% higher costs in resource-limited settings — and the review offers mitigation strategies rather than evidence that those strategies work.

Citation

Mahamad, S., Chin, Y.H., Zulmuksah, N.I.N., Haque, M.M., Shaheen, M., & Nisar, K. (2025). Architecting an AI-Driven Decision Support System for Enhanced Online Learning and Assessment. Future Internet, 17(9), 383.