Automatic Large Language Models Creation of Interactive Learning Lessons

Synthesis: Lin et al. (2025) explore automatic generation of interactive, scenario-based lessons designed to train novice human tutors who teach middle-school mathematics online. The system combines prompt engineering with a Retrieval-Augmented Generation (RAG) approach on GPT-4o, generating lessons in English for three tutor-training topics (Encouraging Students' Independence, Encouraging Help-Seeking Behavior, Turning on Cameras). The key finding is that a task decomposition prompting strategy — breaking lesson generation into segments — produced higher-rated lessons than single-step generation, with a three-segment approach scoring highest. Two human evaluators rated the lessons using a rubric informed by lesson design research (Cohen's κ = 0.72), identifying strengths (well-structured content, time-saving) and limitations (generic Feedback, unclear sections, unreliable references), supporting a hybrid human–AI lesson-generation workflow.

Context and motivation

One-on-one human tutoring is highly effective but difficult to scale due to a scarcity of skilled tutors. Professional development programs train novice tutors, and online scenario-based training has emerged as a promising approach — but manually constructing the lessons at its center is time-consuming and expertise-heavy, making it costly to scale. The authors address this with Large Language Models (LLMs)-based automatic lesson generation.

Method

Lessons follow the SMART tutor-training framework's five-section structure (Title Page, Scenario I, Instruction, Scenario II, Conclusion). The generation pipeline has two steps: a lesson designer retrieves research articles on effective tutoring practices, which then prompt the GPT-4o model (gpt-4o-2024-05-13) via RAG to produce pedagogically grounded content.

To explore optimal prompting, lessons were generated under five segmentation strategies: one segment (whole lesson at once), two, three, four, and five segments — where each segment is generated using prior segments as context. Evaluation used two human coders experienced in tutoring middle-school math, applying a 17-code rubric (Cohen's κ = 0.72), with a third reviewer resolving discrepancies.

Results

The three-segment approach achieved the highest average rating (14.67), while the one-segment approach scored lowest (10.67), indicating that moderate task decomposition improves lesson quality. However, the five-segment approach — the highest level of decomposition — performed slightly worse than three segments, suggesting that excessive decomposition may reduce clarity and pedagogical grounding. Across the rating codes, three-segment generation was particularly strong on feedback and pedagogical grounding.

Human lesson designers identified several strengths: substantial time savings in drafting scenarios and multiple-choice questions, generation of diverse and realistic scenarios, no observed biased or offensive content, and a clear foundation for iterative human–AI collaboration. Limitations included generic feedback that mostly explained the correct answer without targeted explanations of why options were wrong, inconsistent terminology (e.g., "learners" vs. "tutors") and unclear learning objectives, overly long and logically disconnected instruction sections, and non-authentic academic references that persisted even with RAG.

What this means for practice

• Designers. Decompose lesson generation into a small number of sub-tasks — three segments produced the highest-rated lessons in this system — rather than prompting the model for a whole lesson at once.
• Designers. Require human verification of every citation: RAG improved contextual relevance but the model still generated references that did not exist, even at the best segmentation setting.
• Designers. Ask explicitly for per-option rationales, because generated Feedback mostly explained only the correct answer and left the distractors unexplained.
• Designers. Constrain terminology and section length in the prompt: generated lessons mixed "learners", "teachers", and "tutors", and produced instruction sections that were overly long and logically disconnected.
• Researchers. Treat automated generation as a way to scale tutor-training content and test its learning effects directly, since the quality evidence here comes from human ratings of the lessons rather than from learner outcomes.

Limitations

• The quality comparison rests on a very small content set: three tutor-training topics generated under five segmentation strategies, rated by two human coders who reached Cohen's κ = 0.72, with a third reviewer resolving disagreements.
• Lessons were not tried with learners. Study RQ2 collected written questionnaire responses from two lesson designers, and the authors state that an experiment comparing LLM-generated with human-crafted lessons on participants' learning outcomes remains future work.
• Evaluation used a single 17-code rubric adapted from one source — the authors note this and call for comparison with established rubrics in the field — and one model family, gpt-4o-2024-05-13, for all conditions.

Citation

Lin, J., Rao, J., Zhao, S. Y., Wang, Y., Gurung, A., Barany, A., Ocumpaugh, J., Baker, R. S., & Koedinger, K. R. (2025). Automatic Large Language Models Creation of Interactive Learning Lessons. ECTEL 2025.