Reducing Barriers to Academic Support: Evaluating a Course-Specific RAG System for Addressing Help-Seeking Disparities in Higher Education

Synthesis: Fifteen computing students and four academics evaluated Beacon, a course-specific retrieval-augmented generation assistant built from a single programming module's approved teaching materials. Students rated its answers highly relevant to the course (89%), found it clear (88% very or somewhat clear) and intuitive (73.3%), and most said it supported rather than replaced their learning (66.7%). The study's argument is about Help-Seeking: 62.5% of respondents said they sometimes avoided asking for help even when they needed it, and the authors position a private, module-grounded tool as a first rung for students who hesitate to approach a lecturer.

Key Findings

Alignment with course materials was the strongest result. 89% of participants rated the system's responses as highly aligned with course materials, and 56% found outputs consistently logical.
Clarity and usability scored well. 44% reported the system was very clear and a further 44% somewhat clear; 73.3% found it intuitive, 73.35% easy to use, and 66.7% were confident navigating it.
Self-reported learning support was moderate. 53.35% agreed Beacon improved their understanding of programming concepts, 53.3% felt more confident solving problems, and 60% felt more confident tackling difficult topics.
The tool was seen as additive, not a substitute. 66.7% said Beacon supported rather than replaced their learning and 80% agreed it helped, while students still described verifying important information themselves.
The help-seeking baseline was the problem the design targets. 62.5% of respondents said they sometimes avoided asking for help when they needed it and 75% reported anxiety when they did not understand a topic.
Academics valued the refusal to give answers. Staff praised the pseudocode, simple starting structures and links to module materials, and worried that unrestricted AI tools let students "skip that development stage".
The evaluation is small and qualitative at its core. Nine interviews were conducted — five with students and four with academics — and 15 students completed the task scenarios, nine in a two-hour in-person workshop and six remotely.

Grounding answers in the module is the design idea

Beacon retrieves from university-curated teaching materials before generating a response, which is the mechanism the authors test against the known failure mode of general-purpose tools: answers that are inaccurate, insufficiently contextualized, or pitched above a novice's level. Students in this study had experienced that failure directly, describing explanations that exceeded their current understanding and left them more confused. Grounding is therefore not a technical nicety but the feature that makes a tutoring-style assistant usable inside a specific curriculum, and the 89% relevance rating is the students' own version of the same claim.

The tension between access and bypass

Staff interviews carry the counter-argument the paper is careful to preserve: a tool that students turn to before office hours may also be a tool that lets them avoid the productive discomfort of debugging. Academics were reassured that Beacon withholds full solutions and provides scaffolded steps, and the authors frame the target as an intermediary space between independent study and formal support rather than a replacement for teaching staff. That is a design position with a measurable consequence — the system's answers were judged educationally appropriate precisely because they refused to be complete.

Equity as the stated motivation

The framing is about unequal access to support rather than about efficiency. Students who are anxious, uncertain of expectations, or low in confidence seek help less often even where support exists, so the paper's contribution is a design that lowers the social cost of asking. The authors also name the risk that course-specific AI advantages students already able to evaluate generated answers critically, which is why the tool's transparency about its sources matters for equity in AI education and not only for trust.

What this means for practice

Instructors. Ground any course AI assistant in your own approved materials, and require it to scaffold rather than supply solutions; in this study that design choice is what made staff and students accept it.
Software developers. Treat help-seeking hesitation as the target metric. Logging when students choose a private AI channel instead of asking a lecturer tells you whether the tool is broadening access or absorbing contact.
Administrators. A course-specific assistant needs a maintenance owner, since its credibility rests entirely on keeping the retrieval sources aligned with current module materials.

Limitations

One programming module at one institution, with 15 students completing the scenarios and nine interviews, so the study reports perceptions rather than comparative outcomes.
All questionnaire results are self-report usability and confidence ratings; no control condition or learning-outcome measure was collected.
The system was evaluated shortly after introduction, so novelty effects on ratings cannot be separated from the design.
The authors declare no conflict of interest but built the system they evaluated, which is a developer-as-researcher design rather than an independent test.

Connected Concepts

Connected Articles

EduGuard: A Safe RAG-Based LLM Tutor for Programming Education — EduGuard: A Safe RAG-Based LLM Tutor for Programming Education
Retrieval-Augmented Tutoring for Algorithm Tracing and Problem-Solving in AI Education — Retrieval-Augmented Tutoring for Algorithm Tracing and Problem-Solving
AlgoRAG: Retrieval-Augmented Generation for Theoretical Computer Science Education -- A Comprehensive Evaluation Framework for Algorithm Analysis and Complexity Theory — AlgoRAG: Retrieval-Augmented Generation for Theoretical Computer Science Education
Efficiency vs. Effectiveness: Self-Regulated Learning with LLM-Mediated Help-Seeking — Efficiency vs. Effectiveness: Self-Regulated Learning with LLM-Mediated Help-Seeking
Regulating the AI Tutor: Intentions, Help-Seeking, and Self-Regulated Learning in Adolescent GenAI Use — Regulating the AI Tutor: Intentions, Help-Seeking, and Self-Regulated Learning
Warning About AI Fallibility Increases Help-Seeking in an Intelligent Tutoring System — Warning About AI Fallibility Increases Help-Seeking in an Intelligent Tutoring System
Exploring the Design Space of LLM-Based Programming Support in CS Education: A Scoping Review through the Lens of Assistance Governance — Exploring the Design Space of LLM-Based Programming Support in CS Education
The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance — The Effects of Structured LLM-Generated Feedback on Programming Assignment Performance

Citation

Gray, A., & Hobbs, J. (2026). Reducing Barriers to Academic Support: Evaluating a Course-Specific RAG System for Addressing Help-Seeking Disparities in Higher Education. arXiv:2609.21600.

Metadata

Study design	User Study
Discipline	Cs Education
Level	Higher Ed
Intended audience	Instructors, Software Developers
Learning and instruction	Help-Seeking, Scaffolding
Technologies and techniques	RAG (Retrieval-Augmented Generation), Large Language Models (LLMs)
Research methods and evaluation	Usability Research, Mixed-Methods Research
Equity, ethics, and responsible use	Equity
Page kind	Evaluation

Source History Edit Chat