Scaffolding Systematic Reviews in Learning Design and Technology Through Mentoring and AI Integration

Synthesis: Wang et al. (2026) treat systematic reviews (SRs) in Learning Design and Technology (LDT) as intentional learning experiences rather than mere methodological procedures. Drawing on their interdisciplinary team of novice and experienced researchers, they reflect on the realities of each SR stage, showing how scaffolding through mentoring, peer collaboration, and judicious AI integration helped manage ambiguity, foster team consistency, and sustain methodological rigor. Automation tools (chiefly screening) reduced procedural burdens, but interpretive decisions required substantial human oversight — a clear instance of Human-in-the-Loop.

Core Finding

Conducting a systematic review is an inherently iterative, human process that is best learned through practice, supported by mentoring and collaboration. Methodological rigor rests on transparent, predefined, reproducible procedures, yet the practical challenges of interdisciplinary literature — variable terminology, reporting conventions, and study formats — demand collective judgment. Automation advanced most in abstract screening, but data extraction, synthesis, and reporting still depended heavily on human expertise. The authors found that hands-on engagement, mentoring support, and peer discussion were central to sustaining team cohesion and methodological rigor, and that automation should be treated as a complement rather than a substitute for human judgment.

The challenge of interdisciplinary synthesis in LDT

Because LDT spans education, instructional design, psychology, information science, computer science, media studies, and more, its literature varies widely in terminology, theory, and reporting conventions. This complicates locating, integrating, and interpreting studies. The authors frame SRs as a way to reduce conceptual ambiguity and build a shared language across disciplines — an inherently collaborative undertaking that benefits from developmental support for novice researchers.

Navigating ambiguity at each SR stage

• Planning: Choosing a viable topic requires prior familiarity with the literature and assessment of field maturity; developing shared inclusion/exclusion criteria demands iterative discussion, calibration exercises, and concrete decision rules for gray-area cases.
• Execution: Abstract and full-text screening face high volume, sparse and inconsistent abstracts, and diverse document formats. Strategies included conservative screening with "maybe" categories, keyword-exclusion filtering, training in academic article structure and reading strategies, and pairing less experienced researchers with more experienced members during double screening.
• Data extraction: Variable reporting locations and conventions reduced interrater reliability. Structured, multiple-choice coding items, practice rounds, double coding with third-reviewer adjudication, and shared meeting logs supported consistency.
• Reporting: Open-ended and mixed-format data were hard to synthesize; close collaboration between synthesis methodologists and content experts, with topic-modeling triangulation, balanced interpretation with efficiency.

Mentoring as emotional and methodological scaffolding

Mentoring functioned as an integrated support system combining methodological guidance with emotional scaffolding. It normalized uncertainty, created spaces for co-reflection with experienced researchers, and made expert reasoning visible through discussion — supporting both technical competence and confidence. When one-on-one mentoring was not feasible, peer-based discussion groups offered mutual support. This is a strong example of Scaffolding applied to a professional research process, and of human–AI collaboration where people remain the interpretive center.

Balancing automation with human judgment

While ML and Large Language Models (LLMs) tools (e.g., ASReview, SWIFT-Review, Covidence, AIScreenR, MetaMate) reduced procedural burdens such as abstract screening, tasks requiring contextual interpretation, reconciliation of ambiguous reporting, or integrative synthesis remained hard to automate. The authors used automation selectively and verified its outputs, concluding it is best treated as a complement rather than a substitute — a clear Human-in-the-Loop position consistent with viewing Generative AI as a supportive, scaffolding tool that augments rather than replaces human thinking.

Relevance to the knowledge base

This paper significantly contributes to the knowledge base's Scaffolding and Human-in-the-Loop threads by treating a research workflow as an intentional learning experience. It provides practice-informed guidance for novice researchers and interdisciplinary teams, illustrates how Collaborative Learning and mentoring sustain rigor, and models a principled division of labor between automation (procedural burden) and human judgment (interpretive decisions) — of direct relevance to Learning Design and AI-assisted research practice.

What this means for practice

• Researchers. Treat mentoring as methodological infrastructure rather than a courtesy: pair less experienced reviewers with experienced ones during double screening and put scheduled co-reflection on the review timeline, not on the margins of it.
• Researchers. Write decision rules for gray-area cases before screening starts, run calibration exercises on a shared sample of abstracts, and keep a shared meeting log of adjudications so the same judgment is applied consistently across the team.
• Researchers. Assign automation to procedural load only — chiefly abstract screening and keyword-exclusion filtering — and keep data extraction, reconciliation, and synthesis with human reviewers; verify every tool output rather than accepting it.
• Researchers. Budget time for the tasks the authors found hardest to automate: variable reporting locations, mixed-format data, and disagreements between coders, which are resolved through practice rounds, double coding, and third-reviewer adjudication.
• Researchers. If you lead an interdisciplinary team, make expert reasoning visible in discussion. In this account, that — not tooling — is what sustained both methodological rigor and novice researchers' confidence across the review.

Limitations

• The paper is a reflective account of one interdisciplinary LDT team's own review process, not a controlled study. No mentoring or AI-supported condition is compared against a baseline, so the strategies are described rather than tested for effect.
• The field mapping rests on 242 published reviews from 11 top-tier LDT journals (2011 to June 2025), and the topic model behind its eight themes was fitted to titles and abstracts only, so the themes reflect metadata rather than full texts.
• Evidence about automation comes from the authors' own workflow plus cited tool studies (e.g., ASReview, Covidence, MetaMate); no head-to-head tool comparison or accuracy benchmark is reported.
• The authors report that inconsistent reporting across primary studies reduced interrater reliability during data extraction, and they addressed this through practice rounds and third-reviewer adjudication rather than reporting a reliability coefficient.

Citation

Wang, X., Dadashipour, F., Basori, Maeda, Y., & Richardson, J. C. (2026). Scaffolding systematic reviews in learning design and technology through mentoring and AI integration. Educational Technology Research and Development.