On this page

Synthesis: Thirty-four first-year musicology students tried a gamified AI quiz after a lecture on music and the supernatural, and they agreed they understood the topic. What divided them was the format. Engagement, feedback and the comparison with a normal tutorial all sat near the middle of the scale with wide spread, and the students who would recommend the app were not the ones who understood the topic better — they were the ones who enjoyed using it. The most useful evidence sits in the objections: a right-or-wrong quiz cannot host the interpretive argument a music tutorial runs on, and feedback that accepts a copied keyword teaches the wrong lesson. The authors' answer is optional review, not a replacement for teaching.

Key Findings

  1. Students agreed they understood the topic. On a five-point scale the understanding item averaged 3.74, with 23 of the 34 agreeing or strongly agreeing and only one dissenting. It was the single point of consensus in the survey.
  2. Everything about the app divided them. Engagement, enhanced understanding, game features, feedback and perception of AI all clustered near the midpoint with far wider spread than the understanding item, and the two comparative items fell below the midpoint: preference over a traditional tutorial and intention to use something similar again.
  3. Hints were the best-received feature. The optional hint function drew the highest rating of any app feature, with a median of 4 and 20 students agreeing against 4 disagreeing, which suggests that support on demand read as help rather than as surveillance or spoiling.
  4. Recommending the app did not track learning. Twenty-one students would recommend it and 13 would not, and their ratings of topic understanding were close and not statistically distinguishable. The authors read this as students judging the mode of learning rather than the difficulty of the material.
  5. It tracked the experience of using it. The recommending group rated future use, AI perception, enhanced understanding, engagement, feedback and preference over tutorials far higher, and those differences survived correction for multiple comparisons. Recommendation is itself a global judgment, so these are associations among related perceptions rather than independent evidence.
  6. The disciplinary case is in the complaints, not the praise. Nine of the 34 comments touched on disciplinary fit and only one explicitly credited the app with critical thinking and interdisciplinary interpretation. The others objected to correct-or-incorrect judging, to expertise "outsourced" to a machine, to inhibited independent thought, and to feedback that accepted an answer copying the question's keywords.
  7. The authors' design answer is supplement, not substitute. They recommend optional or semi-structured review outside class time, in-class use limited to a brief demonstration or debrief, and open-ended prompts plus feedback that distinguishes surface keyword matching from interpretive understanding.

The intervention and the sample

The study ran in Semester 1, 2025, inside This Is Music, a compulsory first-year unit at the Sydney Conservatorium of Music that treats music as an interdisciplinary field and changes theme weekly. The intervention fell in week 9, when the theme was Music and the Supernatural; students attended a one-hour lecture and then completed the quiz during a one-hour tutorial. Mystic Melodies ran on Cogniti and was powered by ChatGPT-4o. Its design combined short-answer questioning, a quest narrative in which correct answers generated mythic gifts and teaching points while incorrect answers withheld gifts and explained why, optional hint requests for adaptive difficulty, and a formative assessment that concluded after ten questions.

323 students were enrolled in the unit. The intervention and survey ran in six tutorial groups comprising approximately 90 students who were invited to participate, and participation in both was optional. Thirty-four valid responses were analyzed, approximately 37.8% of students invited. No demographic or background variables were collected, so the sample is an anonymous convenience sample from one cohort in one institution.

The instrument was a five-point Likert survey from 1 (Strongly Disagree) to 5 (Strongly Agree), developed for this study rather than adapted from a validated scale, with items derived from the intervention's design features. It covered overall evaluation plus open-ended questions on effective aspects, improvements, and recommendation. Understanding items were treated as self-reported perceptions, not measures of learning gain.

What the numbers show

Topic understanding sits apart from everything else: a mean of 3.74, a median of 4, an interquartile range of 1, and one dissenting respondent. The app-evaluation items cluster nearer the midpoint with much wider spread, and both comparative items fall below it: 11 respondents preferred the app to traditional tutorial instruction and 11 expected to use similar apps in future, against 14 who did not.

Splitting by recommendation sharpens the picture. The 21 who would recommend and the 13 who would not were close on perceived understanding, 3.81 against 3.62, and far apart on everything else: engagement 3.71 against 2.00, feedback 3.76 against 2.15, adaptive hints 3.81 against 2.77, game features 3.48 against 2.08, and preference over traditional tutorials 3.29 against 1.85. Every intervention-related comparison survived correction. The authors caution that recommendation is itself a global judgment, so group differences are associations among related perceptions, not independent evidence, and the understanding item had lower variance to begin with. Still, the direction suggests students were not rejecting the material as too hard; they were judging the mode of learning.

That distinction matters for anyone reading evaluation data. A satisfaction survey quoting the 3.74 mean could call the quiz a success, yet students who understood the topic could still regard it as a poor substitute for a tutorial, and the two items that measure exactly that comparison sit well below the midpoint.

Why the resistance looked disciplinary

The qualitative analysis produced several themes — feedback and teaching points, engagement and gamification, revision and review, AI limitations, and disciplinary fit — but the paper concentrates on the last because it alone offers a potential explanation for the central quantitative pattern. The frame comes from disciplinary-thinking research: Wineburg on historical thinking as a learned practice, Shulman on signature pedagogies, Clarence on the specificity of disciplinary knowledge. On that view, music humanities students learn to connect sound to cultural meaning, judge interpretations, and recognize what counts as evidence, not merely to recall repertoire facts.

Nine of the 34 respondents made comments coded under disciplinary fit, applied both where concerns were explicit and where they could reasonably be inferred, since first-year students may lack the vocabulary to name them. The clearest positive case, Student 28, called the tool "particularly effective for enhancing critical thinking and deepening understanding by offering diverse perspectives on topics like the intersection of music, culture, and the supernatural" — the only comment explicitly linking the tool to critical thinking and interdisciplinary interpretation.

The weight lies in the objections. Student 13 asked for "more opinion based response questions" and for the system to "converse with the response in an engaging way that initiates further inquiries" instead of judging answers correct or incorrect, which the authors read as a mismatch between quiz logic and the humanities tutorial. Student 15 objected to learning "outsourced to a computer that has no understanding of music or art." Student 25 said AI "does not allow you to learn effectively and think independently." Student 27 valued feedback split into "Assessment, Explanation, and Teaching point" but caught the AI accepting an answer that copied the question's key words.

Student 26 supplies the design conclusion in miniature: the tool was "an interesting tool to offer a different approach to revise lecture material," but "I don't think it can replace traditional tutorials." Across these comments, the authors argue, students were evaluating not only usability but whether the interaction fit the inquiry they associate with music humanities — interpretation, dialogue, expertise, independent thought. They also insist the lens is limited: disciplinary fit is one plausible explanation among several, not a demonstrated dominant one.

What this means for practice

  • Instructors: Do not make a quiz app the central or compulsory tutorial activity for everyone. The authors recommend optional or semi-structured review outside class time, with class time reserved for discussion and interpretation, and no more than a brief in-class demonstration or debrief.
  • Designers: Build open-ended and opinion-based prompts, make feedback explicitly interpretive rather than only corrective, and test whether the system can distinguish a copied keyword from an argued answer before students trust its verdicts.
  • Administrators: Report app-experience items separately from perceived topic understanding. A high understanding mean alongside a low rating for comparison with traditional tutorials describes a cohort that felt it had learned but did not endorse the mode, and averaging the two hides the decision you need to make.
  • Researchers: Treat recommendation groups as analytic categories rather than student types, and pair self-report with objective or artifact-based measures, since this design cannot separate the app's effect from the novelty of the activity.

Limitations

  • The data come from one survey, one cohort, one institution, and one activity in one topic.
  • Thirty-four valid responses represent approximately 37.8% of the approximately 90 students invited across six tutorial groups, and participation was optional.
  • No demographic or background variables were collected, including age, gender, and prior experience with AI, so prior attitudes cannot be ruled out.
  • All outcomes are self-reported perceptions collected immediately after the activity, with no objective learning measure and no pre/post comparison.
  • There was no comparison or control condition, so the study cannot say whether other tutorial activities would have produced different responses.
  • The disciplinary-fit evidence is thin — nine of 34 comments, only one explicitly positive — and the authors state it does not show disciplinary thinking was the dominant basis for evaluation; the instrument was not validated.

Citation

Lee, G. S. K., Case, L., Rose, J., Ho, J., & Tran, A. (2026). Disciplinary thinking and AI perception: a case study of undergraduate students’ evaluation of an AI music quiz. Frontiers in Education, 11, 1968048. https://doi.org/10.3389/feduc.2026.1968048

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.