Skepticism vs. Convenience: Physics Students' Perceptions and Use of Large Language Models Before and After Instruction

Synthesis: First-year physics majors at Virginia Tech answered anonymous surveys before and after a reflective lesson on how large language models work and how they affect learning. Skepticism rose sharply — 88% came to believe that LLMs can leave students with a false sense of confidence, up from 58% — and agreement that an LLM outperforms the average physics student fell from 54% to 32%. Use did not follow perception: convenience and deadline pressure remained the dominant reasons for turning to these tools, and the authors describe the result as a standing tension between recognized risk and convenient practice.

Key Findings

1. Instruction raised skepticism about false confidence. 88% of students agreed after the lesson that LLMs can leave them with a false sense of confidence about their understanding, up from 58% before it.
2. Belief in superior capability dropped. Agreement that LLMs are currently better at solving problems than the average physics student fell from 54% to 32% across the lesson.
3. Students expect strong Problem Solving performance. 79% agreed that LLMs can solve introductory physics problems with high accuracy, against 13% disagreement.
4. Dependence was widely acknowledged before instruction. 81% agreed that over-relying on LLMs to complete coursework may make them dependent on the tools later, with 10% disagreement.
5. Convenience and deadlines drove use. 71% agreed convenience motivated their LLM use and 65% cited deadline pressure, both well ahead of reasons grounded in learning.
6. Comfort with machines over people was split, not settled. 42% agreed they felt more comfortable getting help from an LLM than from an instructor or TA, and 38% disagreed; comfort over classmates was 38% agreement against 40% disagreement.
7. The survey itself was small and unlinked. 53 students participated before the lesson and 42 after, and because responses were anonymous the authors flag attrition between the two surveys as a sampling-bias risk.

A reflective lesson changes beliefs faster than behavior

The instructional package had three parts: a problem-solving activity done both with and without the LLM, instruction on the internal functioning of language models as probabilistic token predictors, and a review of research on LLM impacts on learning. The measurable movement was almost entirely on the belief side. That is a useful and honest finding for anyone who assumes an AI Literacy lesson changes use: perceptions shifted toward critical evaluation, while the convenience and deadline motivations that actually drive student AI use were left untouched by a single lesson.

Why convenience is the binding constraint

Deadline pressure and convenience are properties of the course and the week, not of the student's beliefs. A physics course that assigns hard problem sets due Friday while teaching skepticism about LLMs has set up a conflict it has not resolved, and the authors' own framing names the tension directly: students recognize the risks of over-reliance, false confidence, and reduced conceptual understanding and use the tools anyway. The design implication is that self-regulation support has to reach the moment of use — practice structures, hint systems, or problem designs that make struggling productive — rather than relying on belief change alone.

Measuring perception is not measuring learning

Every number in this study is self-report from anonymous surveys, with no linked pre/post pairs and no performance or retention measure. The authors are explicit that attrition creates sampling bias for aggregate change, and they deliberately did not measure whether students actually learned more or less with the tool. The study therefore documents what students say about their use, and its value is in that scope: a well-instrumented picture of a required first-year course's perception profile, which other Physics Education instructors can compare against.

What this means for practice

• Instructors. Pair any lesson on how LLMs work with a change to the conditions of use. Beliefs shifted in this study while convenience-driven use did not, so a policy or task redesign has to carry the behavioral half.
• Course and curriculum designers. Address deadline pressure explicitly, since it was the second strongest motivator reported; alternative submission timing or scaffolded problem sets act on the stated driver rather than on attitudes.
• Learning designers. Build the reflective lesson into the course sequence rather than delivering it once; the belief shift this study measured is the easier half, and repeated contact with how these models actually work is what keeps it available when a deadline arrives.

Limitations

• 53 pre-lesson and 42 post-lesson responses from one required first-year course for physics majors at a single institution, with anonymous surveys that cannot be linked across the two waves.
• All measures are self-reported beliefs and stated use; no observation of actual use, no measure of learning or retention, and therefore no evidence about effects on learning.
• The intervention was a single lesson taught by the course instructors, one of whom was also an undergraduate learning assistant during the study, so researcher and instructor roles overlap.
• The pre and post comparisons are aggregate percentages rather than matched individuals, which the authors identify as a sampling-bias risk.

Connected Concepts

• Large Language Models (LLMs)
• Physics Education
• Self-Regulated Learning
• Metacognition
• Cognitive Offloading
• AI Literacy
• Student-AI Interaction
• Trust Calibration
• Higher Education
• Self-Report Measures
• Student Engagement

Connected Articles

• Pragmatic users and skeptical nonusers: A qualitative typology of ChatGPT adoption in physics education — Pragmatic users and skeptical nonusers: A qualitative typology of ChatGPT adoption in physics
• Exploring Students' Perceptions of Using Generative AI-Assisted Problem Posing — Exploring Students' Perceptions of Using Generative AI-Assisted Problem Posing
• Probing AI-Generated Physics Solutions and Preparing Students to Critique Them — Probing AI-generated physics solutions
• Trust-utility gap in introductory physics education: Students' adoption, domain-specific skepticism, and preferences for AI integration — The trust-utility gap in physics students' use of AI
• Comparing teacher-centered and student-centered agents based on prompt engineering: Effects on learning performance, cognitive load, flow experience, and empathy perception in physics learning — Comparing teacher-centered and student-centered agents based on prompt engineering
• A multi-agent AI classroom based on dual-process reasoning hazards: a pilot with prospective physics teachers — Multi-agent classroom simulation with physics teachers
• Students' Epistemological Beliefs and their Chatbot Preferences in AI-mediated Physics Learning — Physics chatbot use and epistemological beliefs

Citation

O'Brien, J., Ellis, M., Robinson, A., Simonetti, J., & Ramakrishnan, N. (2026). Skepticism vs. Convenience: Physics Students' Perceptions and Use of Large Language Models Before and After Instruction. arXiv:2609.21037.