Chat as Learning: Student-AI Conversations as Discipline-Associated Cognitive Engagement Patterns

On this page

Overview

This paper proposes a "Chat as Learning" measurement paradigm that treats student prompts to AI teaching assistants as observable externalisations of in-progress learning cognition, complementing outcome-based assessment with a process-level signal. Analysing over 60,000 student messages from 116 courses across four universities (Semesters 3 and 4 of the Uedu platform), the authors combine automated Bloom's Taxonomy classification with a within-person, cross-discipline design in which the same students serve as their own controls across courses in different disciplines. The findings show that Student AI Interaction reflects discipline-associated cognitive engagement patterns rather than fixed individual traits, with implications for how Generative AI teaching assistants should be designed and evaluated in Higher Ed.

Key Findings

Approximately 62% of student prompts reflected higher-order cognitive demand (Apply or above) consistently across two semesters and four disciplines (61.4% in Semester 3, 62.8% in Semester 4), consistent with the view that Student Engagement operates at the constructive/interactive end of the ICAP continuum rather than as mere retrieval.
The same students showed measurably different Bloom-level prompt profiles across disciplines: STEM courses were Apply-prevalent (20.8%), Language & Writing courses were Understand-prevalent (31.7%), and Social Science courses were Create-prevalent (33.8%); a Humanities profile (Evaluate-prevalent, 24.4%) was reported descriptively only given a small course base.
Paired within-person comparison found that the same students produced significantly higher proportions of higher-order prompts in Social Science than in STEM courses (pooled n = 16, p < .001), with the same direction of effect observed in two semester-specific samples.
Within-person shifts were largest at the Create level: the same students produced only 7.7%–19.7% Create-level prompts in STEM courses but 37.8%–42.8% in Social Science courses (an 18–35 percentage-point increase), while Understand-level prompts dropped by roughly 14–15 points.
A crossed random-effects GLMM confirmed the Social Science vs. STEM contrast (pooled OR = 1.51, 95% CI [1.33, 1.72], p < .001) and showed that course-level variance in higher-order engagement (σcourse ≈ 0.96) substantially exceeds student-level variance (σstudent ≈ 0.41), re-orienting interventions toward course-level features.
The automated LLM Bloom classifier was validated against human coding (binary LLM–human Cohen's κ = .426–.606, best-pair consensus κ = .753), with disagreement occurring mainly on messages ambiguous to human raters as well, supporting the use of Prompt Engineering traces in learning analytics.

Implications for Practice

AI teaching assistants should be designed with disciplinary context in mind rather than as one-size-fits-all tools — e.g., STEM assistants might foreground procedural Scaffolding and worked-example progression, while Social Science assistants might foreground argument scaffolding and perspective-taking prompts.
Evaluation metrics for AI teaching assistants should account for disciplinary norms: a predominantly Apply-level interaction pattern indicates productive engagement in a STEM context but might suggest insufficient depth in a Social Science context, and can be benchmarked against discipline-specific cognitive-engagement norms.
Because course-level factors (instructor Pedagogy, AI system prompt configuration, assessment structure) dominate student-level differences, the most informative references for an individual course and student are their discipline norm and course rather than the platform mean.
Critical Thinking claims about student AI use benefit from process-level prompt traces: the discipline-associated, within-person patterning of prompts is inconsistent with a uniformly retrieval-only account of how students use AI.

Connected Concepts

Connected Articles

Citation

Chat as Learning: Student-AI Conversations as Discipline-Associated Cognitive Engagement Patterns — Chang, C.-K., & Li, K.-H. (2026). Computers and Education: Artificial Intelligence, 11, 100644.

Source History Edit Chat