Research Article
The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows
Synthesis: Three system properties enable the fallacy via two cognitive mediators:
The Large Language Models (LLMs) fallacy is a cognitive attribution error in which users misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability (∆C). It is independent of output correctness — it persists whether the AI is accurate or hallucinating.
Distinctions from Related Phenomena
| Concept | Focus | LLM Fallacy |
|---|---|---|
| Hallucination | System produces incorrect information | How the user interprets any output as self-generated competence |
| Automation bias | Over-reliance on system during decisions | Self-perception of personal capability derived from outputs |
| Cognitive offloading | Delegating mental effort to tools | Integration of outputs into user's identity and self-evaluation |
| Dunning-Kruger | Internal miscalibration of skill | Specifically AI-mediated; requires tool interaction to emerge |
Mechanisms
Three system properties enable the fallacy via two cognitive mediators:
System Properties:
- Opacity — Users cannot trace how the model constructed the response; division of labor is invisible
- Fluency — Polished, coherent output acts as a metacognitive cue for competence; users infer skill from surface ease rather than generative process
- Interactional immediacy — Rapid response cycles bias toward fast, intuitive judgments over reflective evaluation
Cognitive Mediators:
- Attribution ambiguity — In iterative interactions, the boundary between user contribution and system generation becomes impossible to delineate; authorship is inferred from outcomes
- Cognitive outsourcing — As the system assumes more workload, users engage less with underlying reasoning, weakening self-assessment accuracy
"Capability divergence (∆C) emerges from the interaction of system-level properties (opacity, fluency, immediacy), mediated by attribution ambiguity and cognitive outsourcing."
Manifestations in Education
| Domain | Educational Example |
|---|---|
| Computational | Student produces working code via Copilot but cannot explain logic, debug independently, or adapt to new requirements |
| Linguistic | Student generates fluent essay in a second language but cannot produce comparable prose unassisted |
| Analytical | Student presents structured step-by-step math solution but cannot replicate reasoning when AI is unavailable |
| Creative / Epistemic | Student reads AI summary of a topic and equates access to information with conceptual mastery (illusion of explanatory depth) |
| Professional signaling | Resumes, portfolios, and interview answers reflect ability to prompt LLMs rather than independently acquired expertise |
Relationship to Existing Knowledge Base Concepts
- Metacognition — The LLM fallacy is a metacognitive calibration failure: students cannot accurately monitor their own understanding because fluent AI output creates false fluency signals
- Transfer of Learning — Misattribution undermines transfer because students believe they have mastered material they have merely prompted; the gap between perceived and actual capability manifests as transfer failure
- SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems — SafeTutors' Cognitive (fluency illusion) and Ethical-Epistemic (misrepresentation) dimensions are tutoring-specific expressions of the fallacy
- Self-Regulated Learning — Attribution ambiguity disrupts the self-evaluation phase of SRL, preventing accurate causal attribution and adaptation
What this means for practice
- Instructors. Require an explain-in-your-own-words step before submission and an unaided re-performance of the same task afterward: the misattribution persists whether the AI output is correct or hallucinated, so fluent work is never by itself evidence of the student's own competence.
- Instructors. Name the mechanism out loud with students — polished, fluent output acts as a metacognitive cue for competence — and have them check their own offloading rather than read ease as mastery. The fallacy needs a task that demands domain expertise, a seamless interaction, and fluent output, which is exactly AI-assisted writing, coding, and Problem Solving.
- Assessment designers. Replace output-only grading with process-aware evidence of the human/system split, such as contribution provenance or an in-class unaided demonstration. AI mediation is invisible to both human and automated evaluators, and grades that rise while Transfer of Learning does not weaken what a credential signals.
- Designers. Cut the system properties that enable the fallacy: surface the reasoner's trace to reduce opacity and require iterative user refinement to reduce interactional immediacy, keeping the boundary between user contribution and system generation salient throughout the task.
Limitations
- The framework is conceptual and reports no dataset of its own: the authors present its cross-domain patterns as conceptual and cross-contextual rather than controlled empirical validation, so capability divergence (∆C) is defined rather than measured.
- Its evidence base is secondhand. The mechanisms draw on prior studies (e.g., Nam et al., 2024; Karny et al., 2024), so the framework inherits those samples, designs, and domain limits without adding new observations of learners.
- The paper itself was drafted through a human–AI collaborative workflow, with LLMs used for drafting support, structural refinement, language optimization, and iterative conceptual exploration under the NLD-P prompting framework — a disclosed method that leaves the interpretive analysis non-independent of the technology being theorized.
- The domain illustrations (computational, linguistic, analytical, creative, professional signaling) are assembled from existing literature as cross-contextual patterns; none is tested against unaided performance in a controlled setting.
Citation
Kim, H., Yu, H., & Yi, H. (2026). The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows.