🧠 AI Ed Wiki

📄 Research Article

Let''s Chat: Leveraging Chatbot Outreach for Improved Course Performance

● high · 2026-08-03higher-ed, student-experience, generative-ai, engagement-metrics, rct, rag, llm

Let's Chat: Chatbot Outreach for Course Performance

Meyer, Page, Mata et al. (2026) ran two pre-registered RCTs at Georgia State University testing a non-generative academic chatbot that texted students 2–3 customized nudges per week in large-enrollment online courses. It raised the probability of earning an A or B by 4 percentage points — driven entirely by women in Microeconomics (+7 grade points, +11 pp A/B, −10 pp DFW) — via a task-completion channel (tutoring attendance, homework completion), with no spillover to other courses.

Design

  • Two RCTs (REES-registered) in GSU's Introduction to American Government and Principles of Microeconomics — large-enrollment, asynchronous online courses
  • Half of students got a Mainstay chatbot: 2–3 scheduled text messages/week (due dates, nudges for missing assignments, encouragement), personalized ("Hi FIRSTNAME") and targeted (differentiated by missing-work status); 24/7 AI responses from a curated content knowledge base, TA fallback for unanswered questions
  • Note: the bot is non-generative AI — rule/AI-answered from a pre-programmed knowledge base — which is precisely why it's relevant: minimal hallucination risk at scale

    • Results

  • +4 pp likelihood of earning an A or B, similar across both courses
  • Microeconomics women: +7 grade points vs control women; +11 pp A/B; 10 pp less likely to DFW. No treatment effects for men
  • Mechanisms: treated students more likely to attend university tutoring; suggestive homework-completion gains in Micro (weekly-assignment course); no assignment effects in Government
  • No spillover: no effects on other courses that term or on next-semester enrollment/performance — course-specific gains without (yet) developed study habits
  • 82% of surveyed students recommended continuing the bot and expanding it

    • Why it matters

  • Large-enrollment online courses have documented negative outcomes; low-touch, scalable outreach is one of the few levers that work
  • Contrasts with other low-touch outreach null results (Oreopoulos & Petronijevic 2019) — timing and customization matter; nudges work best on ongoing, dynamic tasks (weekly assignments) rather than one-shot inputs
  • Shows the channel matters: chatbot outreach moved students to human tutoring — a complementarity rather than substitution story
  • A useful counterpoint to GenAI Can Harm Teaching RCT 2026: this is student-facing, non-generative, low-stakes outreach, where AI assistance had positive effects — the harm findings are about generative teacher-facing tools used for delegation

    • Connected Concepts

  • Generative AI
  • Higher Ed
  • Student Experience
  • LLM
  • RAG

    • Connected Articles

  • Beyond Detection Authentic Assessment AI 2025 — Beyond Detection: redesigning authentic assessment in an AI-mediated world
  • GenAI Availability Grades Satisfaction — Generative AI Availability, Grades, and Student Satisfaction at a Large University
  • GenAI Can Harm Teaching RCT 2026 — Generative AI Can Harm Teaching
  • A4l Analytics Pipeline — Generalizing a Highly Configurable Analytics Pipeline to Replicate and Support Educational Research Across Multiple D...
  • Aaai2026 Prompting Literacy K12 — Learning to Use AI for Learning: Teaching Responsible Use of AI Chatbot to K-12 Students Through an AI Literacy Module
  • Academiclaw Student Agent Benchmark — AcademiClaw: When Students Set Challenges for AI Agents
  • Access Not Enough AI Tutoring 2026 — Access is Not Enough: Human Support Improves Engagement with AI Tutoring
  • Adapt Adaptive Lesson Plan Transformer — AdaPT: Adaptive Lesson Plan Transformer for Cross-Regional and Differentiated Instruction
  • Adaptive Pretesting Retention — Do Gains from Generative AI-Enabled Adaptive Pretesting Persist? Evidence from a Retention Study
  • Affective Text Wearable Student Health — A Formative Study of Brief Affective Text as a Complement to Wearable Sensing for Longitudinal Student Health Monitoring
  • Agency Gap AI Writing — The agency gap in AI-supported writing: how reactive and proactive agent designs shape multimodal reasoning
  • Agent Voice Accents K12 Group Learning — Exploring How Agent Voice Accents Shape Human-AI Collaboration in K-12 Group Learning
  • Agentic AI Education Scoping Review — Agentic AI in Education: A Scoping Review of Research Landscape, Capabilities, and the Frontier Agent Paradigm
  • Agentic Education Coding — Agentic Education with AI Coding Assistants
  • Agentic Literacy Debt — Agentic Literacy Debt: A Structural Problem the AI Literacy Field Has Not Yet Named
  • Agents That Teach Incidental Learning — Agents That Teach: Designing Incidental Learning Back into AI-Assisted Software Development
  • Agreement Not Quality LLM Coding Verification — Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not G...
  • AI Adult Learning Guidelines Dis2026 — Guidelines for Designing AI Technologies to Support Adult Learning
  • AI Agents Constructive Conflict Design Education 2026 — Enacting Constructive Conflicts with AI Agents to Enhance Reconsideration among Novice Interaction Designers
  • AI Agents Peer Learning Discourse — When AI Agents Teach Each Other: Discourse Patterns Resembling Peer Learning in the Moltbook Community
  • AI Assessment Scale Reform — A bit of chaos and madness": The AI Assessment Scale and the work of assessment reform
  • AI Assistance Discretionary Feedback — AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education
  • AI Assisted Learning Modes Eeg — An exploratory behavioral and electroencephalographic study of artificial intelligence-assisted learning modes in hig...
  • AI Assisted Se Curriculum Syllabus Analysis 2026 — Mapping the Emerging Curriculum for AI-Assisted Software Engineering via Syllabus Analysis
  • AI Assisted Writing Research Teams — Smaller, Younger, and More Impactful: How AI-Assisted Writing Transforms Research Teams

    • Citation

    Meyer, K. E., Page, L. C., Mata, C., Smith, E., Walsh, B. T., Fifield, C. L., Tyson, M., Eremionkhale, A. E., Evans, M., Frost, S., & Jung, E. E. (2026). Let's Chat: Leveraging Chatbot Outreach for Improved Course Performance. NBER Working Paper No. 35397. NBER