Understanding student perspectives on generative AI chatbots: a human-centred mixed-methods study in higher education

Synthesis: Colbran, Jha, and Schiavone examined how students perceive, use, and evaluate generative AI chatbots in higher education, combining chatbot system analytics with a mixed-methods survey (n=121) around the "Jordan Chatbot," a human-centred pedagogical agent built on GPT-4o within a learning management system. Students held generally positive attitudes toward chatbots and perceived gains in knowledge and understanding while strongly supporting academic-integrity requirements; usage analytics confirmed 24/7 needs, with 36.8% of interactions occurring after hours. AI literacy, rather than general technology experience, was associated with willingness and confidence to use chatbots, and usability (the intrusive pop-up design) was the biggest barrier among non-users. The study recommends human-centred design, explicit AI policies and Assessment labels, staff and student training, and continuous monitoring of errors.

Core Finding

Students see generative AI chatbots as valuable supplementary tools that complement rather than replace human instruction, valuing their Accessibility, rapid responses, and unit-specific support while maintaining robust support for academic-integrity rules — yet adoption hinges more on AI literacy and user experience than on technical capability alone. The significant gap between high general technological literacy and lower AI familiarity, and the finding that users had greater prior chatbot exposure, suggest that targeted training and exposure can close adoption gaps.

Design and Deployment

The Jordan Chatbot was designed as a human-centred pedagogical agent grounded in Constructivist and socio-cultural learning theory, built on GPT-4o and integrating system instructions, a curated knowledge base of disciplinary and institutional materials, and an embedded LMS interface. Situated within authentic disciplinary contexts (Australian criminal law), the design aligns with situated learning theory and Vygotsky's zone of proximal development, positioning the chatbot as a mediating tool that extends, but does not replace, teacher guidance. Transparent policies, ethical parameters, and user control reflect human-centred design principles and responsible AI frameworks, with restrictions on answering assessment-related questions to safeguard academic integrity.

Student Attitudes and Usage

Students reported generally positive attitudes toward chatbots and perceived gains in knowledge and understanding, though they did not see chatbots as producing better results than students could on their own. Intermediate-level English language speakers were more favorable toward chatbot use than advanced speakers, suggesting chatbots particularly support language-learners navigating complex content. Females reported greater prior chatbot exposure and a more positive attitude, while males were more likely to believe chatbots could improve study grades. Usage analytics confirmed the value of 24/7 availability, with over a third of interactions occurring after hours — a key affordance for online and distance learners.

Barriers and User Experience

Five significant reasons for non-use emerged: lack of training, concerns about Trusted/accurate/safe/helpful responses, a preference for asking teaching staff, fear of breaching academic integrity, and no interest. Qualitative analysis of non-user comments (reflexive thematic analysis) surfaced usability as the largest theme (23%), with students disliking the intrusive pop-up design; other themes included preference for humans or search engines, lack of perceived usefulness, AI concerns, unawareness, and time pressure. Users valued helpfulness for information and navigation (24%), availability and accessibility (21%), and subject-specific learning support (18%), while reporting occasional inaccuracies and requesting better integration of reliable links, chat history, and more teacher-like responses.

Relevance to the wiki

This paper gives the wiki rich, real-world evidence on Student AI Interaction and Student Experience with Conversational AI in Higher Ed, showing how AI Literacy — not just general digital proficiency — shapes adoption of Generative AI pedagogical agents. Its attention to Academic Integrity fears and trust connects to the wiki's coverage of reducing AI misuse, while its human-centred design recommendations inform Instructional Design and the conditions under which chatbots equitably complement human instruction.

Connected Concepts

Connected Articles

Citation

Colbran, S., Jha, M., & Schiavone, C. (2026). Understanding student perspectives on generative AI chatbots: a human-centred mixed-methods study in higher education. International Journal of Educational Technology in Higher Education.

View Source View Changes Edit