---
source_url: https://arxiv.org/abs/2508.16659
ingested: 2026-05-08
sha256: 0a9bff57bca97666635d9b18da193c4b992fdcd3eb7d68d69e90523d91aeef47
---

# Enabling Multi-Agent Systems as Learning Designers: Applying Learning Sciences to AI Instructional Design

**Authors:** Wang, Jiayi, Xiao, Ruiwei, Hou, Xinying, Stamper, John
**Published:** 2025/08/20
**arXiv:** https://arxiv.org/abs/2508.16659
**PDF:** https://arxiv.org/pdf/2508.16659
**HTML:** https://arxiv.org/html/2508.16659

## Abstract

K-12 educators are increasingly using Large Language Models (LLMs) to create instructional materials. These systems excel at producing fluent, coherent content, but often lack support for high-quality teaching. The reason is twofold: first, commercial LLMs, such as ChatGPT and Gemini which are among the most widely accessible to teachers, do not come preloaded with the depth of pedagogical theory needed to design truly effective activities; second, although sophisticated prompt engineering can bridge this gap, most teachers lack the time or expertise and find it difficult to encode such pedagogical nuance into their requests. This study shifts pedagogical expertise from the user's prompt to the LLM's internal architecture. We embed the well-established Knowledge-Learning-Instruction (KLI) framework into a Multi-Agent System (MAS) to act as a sophisticated instructional designer. We tested three systems for generating secondary Math and Science learning activities: a Single-Agent baseline simulating typical teacher prompts; a role-based MAS where agents work sequentially; and a collaborative MAS-CMD where agents co-construct activities through conquer and merge discussion. The generated materials were evaluated by 20 practicing teachers and a complementary LLM-as-a-judge system using the Quality Matters (QM) K-12 standards. While the rubric scores showed only small, often statistically insignificant differences between the systems, the qualitative feedback from educators painted a clear and compelling picture. Teachers strongly preferred the activities from the collaborative MAS-CMD, describing them as significantly more creative, contextually relevant, and classroom-ready. Our findings show that embedding pedagogical principles into LLM systems offers a scalable path for creating high-quality educational content.

## Full Text (Extracted from HTML version)

K-12 educators are increasingly using Large Language Models (LLMs) to create instructional materials. These systems excel at producing fluent, coherent content, but often lack support for high-quality teaching. The reason is twofold: first, commercial LLMs, such as ChatGPT and Gemini which are among the most widely accessible to teachers, do not come preloaded with the depth of pedagogical theory needed to design truly effective activities; second, although sophisticated prompt engineering can bridge this gap, most teachers lack the time or expertise and find it difficult to encode such pedagogical nuance into their requests. This study shifts pedagogical expertise from the user’s prompt to the LLM’s internal architecture. We embed the well-established Knowledge–Learning–Instruction (KLI) framework into a Multi-Agent System (MAS) to act as a sophisticated instructional designer. We tested three systems for generating secondary Math and Science learning activities: a Single-Agent baseline simulating typical teacher prompts; a role-based MAS where agents work sequentially; and a collaborative MAS–CMD where agents co-construct activities through conquer and merge discussion. The generated materials were evaluated by 20 practicing teachers and a complementary LLM-as-a-judge system using the Quality Matters (QM) K-12 standards. While the rubric scores showed only small, often statistically insignificant differences between the systems, the qualitative feedback from educators painted a clear and compelling picture. Teachers strongly preferred the activities from the collaborative MAS-CMD, describing them as significantly more creative, contextually relevant, and classroom-ready. Our findings show that embedding pedagogical principles into LLM systems offers a scalable path for creating high-quality educational content.

The emergence of powerful Large Language Models (LLMs) since 2022 has rapidly transformed K–12 education, with surveys showing growing adoption for tasks like lesson planning (Klopfer et al. 2024 ; Tan, Cheng, and Ling 2025 ) . Yet a persistent “prompting gap” limits their effectiveness: while teachers can improve prompt quality with practice, most lack the time or expertise to consistently generate high-quality outputs (Octavio, Argüello, and Pujolà 2024 ) . Without training or advanced tools, LLMs remain difficult to use as true instructional partners.

To address this gap, AI must evolve from passive text generators to active pedagogical partners by embedding learning science principles directly into content generation. We adopt the Knowledge–Learning–Instruction (KLI) framework, which connects knowledge components, learning processes, and instructional principles to support robust learning (Koedinger, Corbett, and Perfetti 2012 ) .

To simulate the complex, multi-step reasoning process in the KLI framework, we employ Multi-Agent Systems (MAS), with core advantages “in its distributed decision-making and problem-solving capabilities” (Li et al. 2024b ) . This system is well-suited to the complex process of instructional design, allowing different agents to assume specific pedagogical roles and complete sub-tasks.

This paper investigates the following research question: Compared to a baseline of simple prompts teachers might use, do multi-agent systems guided by the KLI framework produce higher-quality learning activities? Our contributions are threefold: (1) we design and implement three distinct generative systems (SAS, MAS-Roles, and MAS-CMD) for learning activity generation; (2) we evaluate the outputs through a mixed-methods analysis involving practicing secondary teachers and a theory-grounded LLM-as-a-judge (Li et al. 2024a ) ; (3) we provide a quantitative analysis of not only the pedagogical quality of the outputs but also their computational cost, revealing a critical trade-off between efficiency and quality. This analysis goes beyond simply identifying the “best” system to understanding the practical costs associated with embedding deep pedagogical theory into generative AI, a crucial consideration for the development of scalable and accessible tools for educators.

Since late 2022, K–12 teachers have increasingly adopted LLMs to streamline lesson planning, activity design, and content differentiation. In fall 2023, 18% of teachers reported using AI for teaching (with another 15% having tried it) (Diliberti et al. 2024 ) . By spring 2025, a study of 2232 public-school teachers found that 60% had used AI during the 2024–25 school year, with 32% using it at least weekly; common uses included preparing to teach (37%), making worksheets/activities (33%), and modifying materials to meet student needs (28%) (Ash 2025 ) . Weekly AI users save about 5.9 hours per week, nearly six weeks over a school year. Most teachers report AI improves their work quality (Ash 2025 ) . Beyond single-agent prompting, multi-agent systems (MAS) show promise in K–12 instructional design by coordinating specialized agents to improve lessons, worksheets, and assessments. For instance, EduPlanner uses evaluator, optimizer, and question-analysis agents to create math lessons tailored to student knowledge (Zhang et al. 2025 ) , while the FACET framework simulates diverse learner profiles to generate individualized worksheets judged by teachers as well-structured and appropriate (Gonnermann-Müller et al. 2025 ) . A study shows that generator–critic MAS pipelines can yield high-quality AI literacy MCQs aligned with Bloom’s Taxonomy and usable in classrooms (Wang, Xiao, and Tseng 2025 ) . These systems suggest that MAS can boost both scalability and pedagogical quality, providing richer, context-aware materials than single-agent prompting. Yet current MAS designs are not explicitly guided by learning sciences principles.

Studies caution that novice prompting often produces shallow, misaligned, or inconsistent materials. In mathematics, analyses of ChatGPT-generated tasks found tendencies toward procedural items, gaps in conceptual representations, and occasional inaccuracies, even after prompt variations, suggesting that unguided prompts can underdeliver on cognitive demand (Sapkota and Bondurant 2024 ) . An evaluation of AI-generated curriculum materials characterized some outputs as “a mile high and an inch deep,” with repetitive activities misaligned to specifications (Sawyer and Aga 2025 ) . In foreign-language lesson planning, researchers found high variability across runs and historical–pedagogical biases that can surface without careful prompt design (Dornburg and Davin 2024 ) . More broadly, studies show that without explicit scaffolds (role constraints, exemplars, rubrics), LLMs can produce hallucinations or surface-level plans that require significant teacher editing (Powell and Courchesne 2024 ) . While more schools and districts are providing trainings to teachers about generative AI (Diliberti et al. 2024 ) , simple and zero-shot prompting remains common (Chen et al. 2024 ) , which may be insufficient for generating pedagogically robust materials.

Developed to bridge the cognitive science research and real-world educational practice, the KLI framework provides a structured, theory-driven approach to instructional design (Koedinger, Corbett, and Perfetti 2012 ) . It states that effective instruction requires careful alignment among three core elements: Knowledge Components (KCs), which are the goals of learning (e.g., concepts, skills, facts); Learning Processes, which describe how learning occurs (e.g., memory and fluency building, induction and refinement, understanding and sense-making); and Instructional Principles, which are the specific methods used to facilitate those learning processes (e.g., spacing and testing, worked examples, prompted self-explanation) (Koedinger, Corbett, and Perfetti 2012 ) . By using the KLI framework to guide the reasoning of our multi-agent systems, we are explicitly grounding our AI systems’ design in a well-established, evidence-based instructional theory that aims to produce robust student learning. This helps us create a more principled and pedagogically-informed instructional design process. Recent work demonstrates how to apply learning sciences principles to generative AI, using the KLI framework as a guide to map knowledge types like debugging to specific tutoring protocols to elicit more effective, learning-oriented responses from the model (Xiao et al. 2025 ) .

The LLM-as-a-Judge paradigm refers to the use of a large language model to assess and score outputs, a technique being applied across various domains, including education (Li et al. 2024a ) . This methodology comes with challenges. LLMs are known to exhibit systematic biases, including a preference for longer outputs, a tendency to favor the first option in a comparison, and a potential preference for outputs generated by models from the same developer family (Gu et al. 2025 ) . When carefully designed, however, this approach offers significant advantages in scalability and speed over traditional human evaluation, and studies have shown that LLM judges can achieve high levels of agreement with human raters. In education-specific contexts, GPT-4-as-judge has shown moderate agreement with human raters when scoring feedback quality with analytic rubrics, though with a positivity bias that requires prompt and aggregation controls (Koutcheme et al. 2024 ) . With the benefits and limitations of LLM-as-a-judge in mind, our study uses this method as a complement for expert evaluation.

The core of this study is designing, implementing, and comparing three systems for generating K–12 learning activities

The Single-Agent System (SAS) was designed as a baseline to represent a naive, non-expert interaction with an LLM. Prior research studies show that non-experts in prompt engineering within educational contexts often rely on simple requests, often consisting of a single verb, and use basic “copy-paste” strategies to provide contextual information (Octavio, Argüello, and Pujolà 2024 ; Tassoti 2024 ) . This SAS design able the measurement of the “value-added” by more sophisticated, theory-informed architectures. Its prompts are constructed by appending contextual information: the subject domain, grade level, standard alignment, and the learning objective, to the phrase “Generate learning activity”.

The Role-Based Multi-Agent System (MAS-Roles) is guided by the KLI theoretical framework. Our system operationalizes this theory through a sequential pipeline of specialized agents, representing a centralized, task-decomposition multi-agent architecture where a complex problem is broken down into manageable subtasks. This system does not merely use concepts from the KLI framework as input, but embodies the analytical process of the framework in its very structure. The system functions as a five-stage automated and iterative pipeline, where the output of each agent serves as the primary input for the subsequent agent. KC Agent: Receives the subject domain, grade level, standard alignment, and learning objective and identifies the core knowledge components (KCs), such as facts, concepts, principles, or procedures. Learning Process Agent: Analyzes the KCs to determine the most appropriate learning process. Instructional Principle Agent: Based on the selected learning process, this agent chooses a relevant instructional principle. Design Agent: Synthesizes the outputs from the previous three agents to generate a complete learning activity. Feedback Agent: Checks the generated activity for coherence and alignment with the initial contextual information, and decides if more iterations and modifications are needed.

The Multi-Agent System with Conquer and Merge Discussion (MAS-CMD) (Wang et al. 2024 ) implements a more dynamic, collaborative architecture designed to simulate a professional discussion. Initial Generation: Three distinct agents are instantiated. Each is given one of the five “teacher personas”: Behaviorist, Constructivist, Aesthetic, Ecological, Integrated Social-Emotional (McConnell, Conrad, and Uhrmacher 2020 ) , for curriculum design to encourage diverse outputs. All three agents receive the same KLI framework guidance as the MAS-Roles system and independently create a draft learning activity. Collaborative Discussion: The three agents engage in a structured, multi-turn dialogue. Each agent presents its draft and asks the other two agents for feedback, then revises its draft based on that feedback. Final Selection: A final decision agent reviews the entire discussion transcript and the three revised drafts. It then chooses the most appropriate one and outputs the final learning activity.

A corpus of 10 learning objectives, 5 from Math and 5 from Science, was selected from the human-curated lesson plan library of the Oak Ridge Institute for Science and Education (ORISE) (Oak Ridge Institute for Science and Education 2025 ) . ORISE is a U.S. Department of Energy institute, providing a repository of high-quality, federally supported STEM resources for educators. The learning objectives selected from this source ensures that the generation tasks are grounded in authentic, curriculum-aligned educational goals. These objectives include subject domain, grade level, and standard alignment information.

Each of the three systems (SAS, MAS-Roles, MAS-CMD) was used with the gemini-2.5-flash model to generate a learning activity for each of the 10 objectives, resulting in a total of 30 unique learning activities for expert human evaluation. In addition, an extra 30 learning activities were generated using each of the following models: gemini-2.5-pro, gemini-2.0-flash, and gemini-2.0-flash-lite, yielding 90 additional learning activities. Combined with the 30 generated using gemini-2.5-flash, all 120 learning activities were evaluated using the LLM-as-a-Judge methodology. The models were selected to represent a range of model capabilities. Default temperature and top-p settings were used throughout the generation process to encourage variation in the outputs.

We recruited 21 US-based secondary math (N=12) and science (N=9) teachers via Prolific to serve as domain experts in evaluating the quality of the generated learning activities under all three conditions. Pre-screening criteria required participants to be located in the United States and teach grades 7–12 mathematics or science, thereby ensuring relevant subject-matter expertise. Each participant received $4–6 in compensation for completing all required tasks, which took approximately 15–30 minutes.

To ensure a rigorous and standardized evaluation, we adapted the Quality Matters (QM) K–12 Rubric (Table 1 ), focusing on four review standards from General Standard 5 (Learning Activities and Learner Interaction) (Quality Matters 2025 ) . Rather than asking evaluators to just assign numbers, we presented multiple-choice, plain-language descriptors that correspond to the rubric’s performance levels. Each item was aligned to the 0–3 scale used in the QM rubric (0–2 for 5.4C), and evaluators selected the descriptor that best matched the activity; we then coded responses to the associated numeric level for analysis. For example, for Standard 5.1C, the options were framed as: clearly aligned to objectives and standards and directly supportive of achieving them (3 points), generally aligned but with limited or inconsistent support (2 points), alignment is weak or unclear (1 point), and does not support the stated objectives or standards (0 point). Similar descriptions were provided for the other criteria. This rubric was used in both human and LLM evaluation.

To enable a transparent and reliable LLM-as-a-judge process, we created the Integrated Learning Sciences Evaluation Rubric (Table 2 ), which synthesizes classic instructional design and learning sciences. The Foundational Alignment & Clarity domain draws on Constructive Alignment and Backward Design to judge coherence among outcomes, activities, and assessments (Biggs, Tang, and Kennedy 2022 ; Wiggins and McTighe 2005 ) . Cognitive Complexity & Rigor operationalizes the revised Bloom’s Taxonomy to locate the primary cognitive demand (Anderson and Krathwohl 2001 ) . Instructional Scaffolding & Process is grounded in Gagné’s Nine Events of Instruction, emphasizing activation of prior knowledge, clear guidance, structured practice, and feedback (Gagne et al. 2005 ) . Learner Engagement & Motivation and Inclsivity & Accessibility are guided by Universal Design for Learning (UDL), providing multiple means of engagement, representation, and action/expression, as well as motivational supports (CAST 2024 ) . Together, these five domains yield a 17-criterion rubric that evaluates the quality of learning activities.

We analyzed the data using a mixed-methods approach. For human expert ratings, each rater evaluated all three systems (SAS, MAS-Roles, MAS-CMD) on QM criteria 5.1 C–5.4 C and a total score. Within-subject differences were tested using repeated-measures ANOVAs (factor: System), and when the omnibus test was significant ( α = .05 \alpha=.05 ), Holm-adjusted paired t t -tests were conducted. Non-parametric tests (Friedman and Wilcoxon) were used as robustness checks. Inter-rater consistency was quantified using Fleiss’ κ \kappa . For LLM-based automatic evaluations on the QM and Integrated Learning Sciences rubrics, we report descriptive statistics (means and standard deviations) only. Operational efficiency metrics (time, tokens, requests) and self-reported AI tool usage were also summarized descriptively to contextualize the results. All quantitative analyses were performed using Python. For the qualitative analysis, written feedback from the K–12 instructors was analyzed using thematic analysis (Braun and Clarke 2006 ) to identify recurring patterns and themes related to the perceived strengths and weaknesses of the activities generated by each system.

One participant’s response was excluded from analysis due to providing off-topic written feedback and completing the task in an unusually short time, indicating insufficient engagement. For evaluation, we relied on data from the remaining 20 participants (11 Math, 9 Science).

Self-reported AI usage patterns (Fig. 1 ) indicate that familiarity is concentrated in general-purpose chatbots. Several teachers reported using ChatGPT (and to a lesser extent Gemini) weekly or daily, whereas most domain-specific tools (e.g., Curipod, Diffit, Brisk Teaching) were rarely or never used. This distribution contextualizes the human ratings: evaluators were more accustomed to general chatbots than to specialized K–12 authoring tools.

Computational costs vary across system designs. (Table 3 ). SAS was fastest and lightest (25 ± \pm 16 s; ∼ \sim 3.4k tokens; one request). MAS-Roles required roughly 3 × 3\times the time and ∼ \sim 7 × \times the tokens, while MAS-CMD traded the highest rubric scores for substantially greater latency and usage (272 ± \pm 168 s; ∼ \sim 71.6k tokens; 13 requests). These results quantify a quality–efficiency trade-off across design strategies.

ANOVA was conducted to compare expert evaluations of the three instructional systems across four criteria (5.1 C–5.4 C) and a composite total score. Each of the twenty experts provided ratings for all three systems on each criterion, enabling within-subject comparisons.

Descriptive analyses indicated that MAS-CMD consistently received the highest average ratings across all four criteria and the total score, whereas MAS-Roles tended to receive the lowest scores. For example, the mean total score was M = 10.05 M=10.05 , 95% CI [9.42, 10.68] for MAS-CMD, compared with M = 9.55 M=9.55 , 95% CI [8.92, 10.18] for SAS, and M = 9.05 M=9.05 , 95% CI [8.28, 9.82] for MAS-Roles. These were computed using the standard confidence interval formula:

ANOVA revealed a statistically significant effect of system on criterion 5.2, F ​ ( 2 , 38 ) = 4.75 F(2,38)=4.75 , p = .014 p=.014 , η p 2 = .20 \eta_{p}^{2}=.20 , indicating a large effect size. Post-hoc pairwise comparisons with Holm correction showed that MAS-CMD was rated significantly higher than MAS-Roles ( p = .050 p=.050 ), with no other significant pairwise differences. For criteria 5.1, 5.3, and 5.4, the omnibus tests were non-significant (all p > .20 p>.20 ). For the total score, the omnibus ANOVA did not reach conventional significance, F ​ ( 2 , 38 ) = 2.66 F(2,38)=2.66 , p = .083 p=.083 , although the non-parametric Friedman test suggested an overall difference across systems, χ 2 ​ ( 2 ) = 6.24 \chi^{2}(2)=6.24 , p = .044 p=.044 . This statistic was computed using the Friedman test formula:

where n n is the number of raters, k k is the number of systems, and R j R_{j} is the total rank for system j j . However, post-hoc Wilcoxon signed-rank tests with Holm correction did not reveal any pairwise comparisons that were statistically significant.

To assess the consistency of expert judgments, we computed Fleiss’ κ \kappa across the twenty raters for each criterion and for the combined scores. Agreement was low overall: criterion 5.1 yielded κ = 0.047 \kappa=0.047 (slight agreement), criterion 5.2 κ = 0.053 \kappa=0.053 (slight), criterion 5.3 κ = − 0.030 \kappa=-0.030 (poor), and criterion 5.4 κ = − 0.043 \kappa=-0.043 (poor). When all criteria were aggregated, the overall κ \kappa was − 0.016 -0.016 , indicating agreement below chance level. These results demonstrate that expert evaluations were highly inconsistent, limiting the reliability of the observed effects.

The SAS produced relevant lesson plans but often lacked the polish, completeness, and innovation that teachers desire. Educators praised the SAS for generating relevant content, such as a lesson on screen time, and for creating engaging activities that positioned students as “detectives”. Some of its outputs were considered “a really solid lesson plan” that could serve as a “springboard” for a teacher’s own ideas. However, SAS was also criticized for a lack of innovation and for failing to provide complete, classroom-ready packages. Teachers noted that student handouts “would need a lot of editing” and expressed a desire for a “print-ready worksheet”. Other common issues included pedagogical and structural flaws, such as a “confusing” warm-up activity, poor pacing where the “timing seems really far off,” and lesson designs that felt too “rigid” for the classroom.

The MAS-Roles system was often seen as a step up from the SAS, producing well-structured lessons, but was sometimes perceived as generic or unresponsive to specific requests. The system was frequently commended for its strong pedagogical design, which included a “step by step approach” and started with “investigation instead of direct instruction”. Teachers found the concepts, like “Geometry Detectives,” to be fun and engaging, and they appreciated the inclusion of both formative and summative assessments within the lesson plans. Despite these strengths, feedback indicated the MAS-Roles system could feel “generic and impersonal,” with one teacher remarking that a lesson reminded them of existing curricula. A significant point of failure was its occasional inability to follow instructions, such as a request for differentiation that was absent from the final plan. Some of the generated activities were also seen as having “very minimal interaction between students,” limiting their collaborative potential.

The MAS-CMD system received the most enthusiastic praise, consistently lauded for its creativity, real-world context, and comprehensive, teacher-friendly resources. Creativity was a standout feature of the MAS-CMD, with teachers describing its ideas as “fantastic” and “so creative”. The system excelled at creating “a strong real-world context,” such as an “urban planning theme” that made geometry “meaningful” for students. It was also highly valued for providing a complete package for educators, including a “worksheet and exit ticket” which one teacher called “Super helpful,” and “teacher dialogue” that was deemed “very helpful” for new teachers. The primary critiques of the MAS-CMD were practical in nature. Some lessons were described as being “a lot to unpack” and having potential “pacing issues,” suggesting they might need to be “broken up into some days” to be implemented effectively. Other minor feedback mentioned the need for more explicit guidance on the use of classroom tools or assessment rubrics.