AIEd Knowledge BaseAIEd KB
On this page

Research Article

Enhancing enthusiasm for STEM education with AI: Domain-specific chatbot as personalized learning assistant

added 2026-09-16 · updated 2026-09-19

Synthesis: Rücker and Becker-Genschow (2025) report a cluster-randomized controlled trial of ADA, a domain-specific generative AI chatbot customized for secondary Math Education, compared with conventional differentiation materials in STEM Education. 195 ninth-grade students in German schools worked through the Heron method for estimating square roots in a single lesson with a pre-post assessment. Students accepted ADA strongly on every Technology Adoption Models dimension and showed significantly higher situational interest in the chatbot condition, while learning performance, emotional responses and cognitive load moved in the expected direction without reaching significance. The authors frame the performance results as trends rather than confirmed gains and conclude that ADA enhanced motivational engagement more than immediate achievement.

Key Findings

  1. The trial involved 195 ninth-grade students (91 female, 104 male) from secondary schools in Germany, with a mean age of 14.33 years (SD = 0.67); the experimental group comprised 102 students (41 female, 61 male) and the control group 93 students (50 female, 43 male).
  2. Sample size planning used G*Power with an effect size f of 0.2, alpha of 0.05 and power of 0.8, producing a target of 200 participants for detecting medium between-group effects.
  3. Technological acceptance was high across all four TAM constructs: perceived ease of use M = 3.44 (SD = 0.83), perceived usefulness M = 3.03 (SD = 0.81), attitudes toward use M = 3.01 (SD = 0.85) and perceived competence M = 3.09 (SD = 0.83), all above 3.0 on a 4-point scale.
  4. Situational interest was significantly higher in the chatbot condition (M = 2.63, SD = 0.51) than in the control condition (M = 2.43, SD = 0.62); a Mann-Whitney U test gave p = 0.00005 and Cohen's d = 0.63, a medium positive effect.
  5. Emotional responses were directionally favorable but non-significant: positive-activating emotions averaged M = 2.37 (SD = 1.03) in the experimental group versus M = 2.10 (SD = 0.96) in the control group (p = 0.053), and negative-deactivating emotions averaged M = 2.45 (SD = 1.15) versus M = 2.61 (SD = 1.15) (p = 0.265).
  6. Cognitive load increased slightly rather than decreasing: intrinsic load M = 2.75 (SD = 1.01) versus M = 2.63 (SD = 0.97) (p = 0.299) and extrinsic load M = 2.62 (SD = 0.98) versus M = 2.38 (SD = 1.02) (p = 0.060), both non-significant, contradicting prior findings of load reduction.
  7. Repeated-measures ANOVA on learning performance found a significant main effect of time, F(1194) = 145.76, p < 0.001, ges = 0.22, and of group, F(1194) = 9.26, p = 0.003, but no significant group by time interaction, F(1194) = 2.84, p = 0.094; the paper therefore treats the performance advantage as a trend rather than a demonstrated gain.
  8. The control group entered the intervention with higher prior knowledge (pre-test M = 8.61, SD = 8.28 versus experimental M = 3.56, SD = 4.18), yet the experimental group improved more from pre-test to post-test (M = 12.42, SD = 12.72 versus M = 9.38, SD = 12.51); these baseline differences, confirmed on all evaluable pre-test variables at p < 0.05, motivated the use of RM-ANOVA.
  9. At subtask level the application task A3 showed the largest descriptive gap (experimental M = 9.44, SD = 10.37 versus control M = 6.68, SD = 11.31) and came closest to an interaction, F(1194) = 3.18, p = 0.076, while the transfer task A4.2(a) also separated the groups descriptively (M = 0.49, SD = 1.42 versus M = 0.29, SD = 1.03).
  10. Every subtask produced a significant time effect (all ps < 0.001), with significant group differences for A1 (p = 0.001), A3 (p = 0.022) and A4.1 (p = 0.005), so gains from pre-test to post-test were general while group separation stayed partial.
  11. Measurement models were validated with exploratory and confirmatory factor analysis: the four-factor TAM model fit almost perfectly (χ²(98) = 87.28, p = 0.773; CFI = 1.00; RMSEA = 0.00; SRMR = 0.06) and did not differ significantly from a three-factor solution (Δχ² = 4.68, Δdf = 3, p = 0.130), so the theoretically consistent four-factor model was retained.
  12. The emotion, cognitive load and situational interest instruments each resolved into two factors, while a unidimensional situational interest model was adopted despite a statistically better two-factor fit, which the authors attribute to methodological artifacts rather than conceptual structure.
  13. Interrater reliability for the two open-ended items, 4.2(a) and 4.2(b), scored by two independent raters on depth of understanding, factual accuracy and content precision, was excellent: Cohen's weighted kappa with quadratic weighting ranged from κ = 0.921 to 0.974 for composite scores and κ = 0.816 to 1.000 for individual dimensions.
  14. Constructs were measured with established instruments adapted to the context and translated into German where needed: seven items for situational interest, five for emotions, five for cognitive load, and four performance items spanning reproduction, application and transfer.
  15. The authors summarize the asymmetry directly: ADA proved more effective at enhancing motivational engagement than at producing immediate performance gains, attributing this to the single-lesson exposure, the dual-task demands of using a chatbot while working with paper materials, and the limited sample.

Study Design & Method

The study used a quasi-experimental controlled intervention with cluster randomization: because of technical constraints, existing class groups were assigned intact to either the chatbot or the control condition, which the authors argue prevented contamination between conditions and strengthened external validity. Both groups received identical paper-based materials on the Heron method of estimating square roots, scaffolded from prerequisite knowledge (Worksheet 1) through geometric exploration (Worksheet 2), independent application (Worksheet 3) and critical optimization (Worksheet 4); the experimental group additionally accessed the chatbot on school laptops, while the control group used static help cards.

ADA is a Custom Generative Pre-Trained Transformer built on GPT-4 via a privacy-compliant API from the German school platform fobizz, with anonymized data transmission for GDPR compliance. It uses Socratic dialogue rather than direct answers, combines Prompt Engineering with retrieval-augmented generation, draws motivational elements from Self-Determination Theory, retrieves from a curated knowledge base covering the curriculum, the intervention materials and background on Ada Lovelace and the Heron method, and calls a Wolfram Alpha plugin to verify student calculations.

Data collection spanned three scheduled mathematics lessons: a single lesson for the pre-test of demographics and prior knowledge, then a double lesson for the intervention followed immediately by the post-test. Teachers followed detailed lesson plans and received a structured online briefing before the study. Assessment used four-point Likert scales throughout, with acceptance, emotions, interest and cognitive load measured only after the intervention. Analyses were run in R (Version 4.4.2), using non-parametric Mann-Whitney U tests for group comparisons after Shapiro-Wilk tests indicated non-normality, and repeated-measures ANOVA with Type III sums of squares for pre-post analysis. The study received ethical approval from the University of Cologne's Faculty of Humanities Ethics Committee, with written informed consent from school administrators, parents and students.

Research agenda

The authors argue that domain-specific custom chatbots can provide targeted support for STEM learning processes, but that the decisive variable is didactic integration rather than the model itself. Their priority for future work is replication of the observed trends with substantially larger samples, followed by testing transferability to a wider range of mathematical and scientific contexts and across educational stages. Because the interest gain did not convert into a significant performance gain within a single lesson, they call specifically for longitudinal designs that show whether motivational engagement translates into achievement over multiple lessons, and whether the short-term performance trends strengthen or fade.

What this means for practice

  • Instructors. Expect a motivation gain, not an immediate achievement gain, from a single chatbot lesson: situational interest was significantly higher in the chatbot condition (M = 2.63, SD = 0.51 versus M = 2.43, SD = 0.62; d = 0.63) while the group-by-time interaction for learning performance was not significant (F(1194) = 2.84, p = 0.094).
  • Instructors. Use a domain-specific bot built on Socratic questioning and curriculum-grounded retrieval instead of a general-purpose chatbot, and keep checking what it says — the authors state hallucinations remain possible even with RAG (Retrieval-Augmented Generation) safeguards, and inconsistent terminology or misleading questioning can still occur, so teacher oversight and an up-to-date curated database are prerequisites. Building AI Literacy among staff and students, and giving educators the competence to configure and curate the bot for their own pedagogical objectives, is what the authors treat as the condition for the bot's benefit.
  • Instructors. Keep the paper worksheets in the lesson rather than replacing them with the bot, and plan a sequence of lessons before expecting achievement effects: the authors attribute the absent performance gain to the single-lesson exposure and to the dual-task demand of working with a chatbot alongside paper materials.
  • Designers. Have the bot question rather than answer, and verify student calculations against an external tool as ADA did with its Wolfram Alpha plugin, while keeping the curated knowledge base current as the curriculum changes.
  • Researchers. Replicate before scaling: the authors' stated first priority is replication of the observed trends with substantially larger samples than the 195 ninth-grade students in this one German learning context.

Limitations

  • The intervention was brief, which the authors accept may have limited the potential for significant effects to emerge, and only immediate short-term effects were measured, precluding any assessment of long-term outcomes.
  • The sample of 195 ninth-grade students came from one specific learning context and learning group, restricting generalizability.
  • The study analyzed validated educational constructs rather than usage metrics, and the authors acknowledge that detailed interaction patterns would have given complementary insight into how students actually used the chatbot.
  • Novelty effects cannot be entirely ruled out, though the authors note that the situational interest finding suggests effects extending beyond mere technological curiosity.

Connected Concepts

  • STEM Education — the subject domain the chatbot ADA was customized for
  • Math Education — secondary mathematics as the intervention context and learning content
  • RCT — cluster-randomized controlled trial design in authentic classrooms
  • Technology Adoption Models — framework used to measure acceptance of ADA, with high scores on all constructs
  • Personalized Learning — the promise of tailored, real-time support that motivated the chatbot design
  • Socratic Method — ADA's questioning strategy in place of direct answers
  • RAG (Retrieval-Augmented Generation) — retrieval-augmented generation used to ground ADA in curated curriculum content
  • Prompt Engineering — iterative system-prompt refinement for accuracy and age appropriateness
  • Motivation — situational interest and emotional responses as the affective outcome cluster
  • Student Engagement — situational interest as the one clearly significant affective gain
  • Cognitive Offloading — the small non-significant increase in intrinsic and extrinsic load
  • Transfer of Learning — the optimization and transfer subtasks that separated the groups descriptively

Connected Articles

Citation

Rücker, C. R., & Becker-Genschow, S. (2025). Enhancing Enthusiasm for STEM Education with AI: Domain-Specific Chatbot as Personalized Learning Assistant. Computers and Education Open, 9, 100315.

Metadata

DisciplineStem Education, Math Education
LevelSecondary, K 12
Intended audienceInstructors, Researchers, Instructional Designers
Learning and instructionMotivation, Socratic Method, Student Engagement
Technologies and techniquesPersonalized Learning, RAG (Retrieval-Augmented Generation), Technology Adoption Models
Research methods and evaluationRCT, Quantitative Research
Foundations of AI in educationCognitive Offloading
SourceHistoryEditChat

Embed this page

Copy the code below to embed a chromeless version of this page in a learning management system or other website. The embedded view hides the site header, navigation, and footer.