AI Ed Wiki logoAI Ed Wiki
Use with AI

📄 Research Article

The Impact of an LLM-Based Educational Agent on Learning Achievement, Cognitive Dynamics, and Student Perceptions in Computer Science Education

● high · 2026-08-22 · llm, agentic-ai, cs-education, intelligent-tutoring, cognitive-diagnosis, cognitive-offloading, student-ai-interaction, higher-ed, learning-analytics

Synthesis: This large-scale quasi-experiment (N = 313 sophomores, four authentic classes, four weeks) evaluated DBagent — a domain-specific LLM-based educational agent for an undergraduate database course with tool use, memory, and goal-directed reasoning. The agent-enriched environment significantly improved learning achievement, but lag sequential analysis of interaction logs revealed a distinctive cognitive profile: high-frequency lower-order engagement (Remember/Understand, ~54.5%) driven by psychological safety, organized around a "Query-Evaluation-Query" verification loop — with only 3.92% of interactions reaching higher-order cognition. SEM confirmed positive perceptions sustain engagement via satisfaction.

Key Findings

  1. Improved learning achievement. The agent-powered context significantly outperformed traditional instruction; the top experimental class beat the control on both tasks (Z = 3.49, Z = 5.15, p < .001), with the largest effect on complex problem-solving (Task 2).
  2. A shift from social inhibition to psychological safety. Student-agent interactions were dominated by lower-order cognitive activity (~54.5%) because the agent provided a judgment-free environment that encouraged Help Seeking — interpreted as psychological safety rather than mere dependency.
  3. The "Query-Evaluation-Query" verification loop. LSA identified a significant QR-EA-QR loop: students offload recall/understanding to the agent, then transition into Evaluation of its output — an "offload-evaluate cycle" distinct from the linear confusion-to-understanding path of human-instructor interaction.
  4. But lower-order lock-in. Strong self-transition loops within lower-order states (Understand z = 49.08; Application z = 51.48) show learners get "locked" in routine processing, with only 3.92% reaching higher-order cognition — attributed to the agent's unwavering compliance lacking pedagogical friction.
  5. Perceptions drive engagement via satisfaction. SEM confirmed learners' positive perceptions of the agent promoted sustained engagement through the mediating role of satisfaction.
  6. The "prompt engineering gap." Efficacy was moderated by domain-specific digital readiness — a Geoscience-major class underperformed CS cohorts on Task 2, suggesting non-technical students need targeted scaffolding to bridge the Prompt Engineering gap.

Implications

This study provides empirical evidence for Intelligent Tutoring and agent-based learning: autonomous LLM agents can improve achievement and reduce social inhibition, but their psychological-safety advantage comes with a cognitive offloading risk — lower-order tasks are offloaded and students can become locked in routine processing without Scaffolding that introduces productive struggle. The finding that only ~4% of interactions reach higher-order cognition echoes the wiki's teach-vs-solve and Tutoring Specific Vs General AI evidence: agent compliance must be designed with pedagogical friction.

The offload-evaluate cycle and lower-order distribution connect directly to Cognitive Diagnosis (evaluating what students actually process) and Cognitive Offloading. The psychological-safety mechanism and the prompt-engineering gap inform AI Literacy and Student AI Interaction — and argue for building verification and critical-evaluation scaffolds into agent design.

Connected Concepts

Connected Articles

Citation

Li, X., Liu, Z., Jiang, S., Chen, J., & Chen, W. (2026). The impact of an LLM-based educational agent on learning achievement, cognitive dynamics, and student perceptions in computer science education. International Journal of STEM Education, 13, 51.