📄 Full text: arXiv:2604.14807 · local
The LLM fallacy is a cognitive attribution error in which users misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability (∆C). It is independent of output correctness — it persists whether the AI is accurate or hallucinating.^kim-llm-fallacy-misattribution-2026
Distinctions from Related Phenomena
| Concept | Focus | LLM Fallacy |
|---|---|---|
| Hallucination | System produces incorrect information | How the user interprets any output as self-generated competence |
| Automation bias | Over-reliance on system during decisions | Self-perception of personal capability derived from outputs |
| Cognitive offloading | Delegating mental effort to tools | Integration of outputs into user's identity and self-evaluation |
| Dunning-Kruger | Internal miscalibration of skill | Specifically AI-mediated; requires tool interaction to emerge |
Mechanisms
Three system properties enable the fallacy via two cognitive mediators:
System Properties: 1. Opacity — Users cannot trace how the model constructed the response; division of labor is invisible 2. Fluency — Polished, coherent output acts as a metacognitive cue for competence; users infer skill from surface ease rather than generative process 3. Interactional immediacy — Rapid response cycles bias toward fast, intuitive judgments over reflective evaluation
Cognitive Mediators:
- Attribution ambiguity — In iterative interactions, the boundary between user contribution and system generation becomes impossible to delineate; authorship is inferred from outcomes
- Cognitive outsourcing — As the system assumes more workload, users engage less with underlying reasoning, weakening self-assessment accuracy
"Capability divergence (∆C) emerges from the interaction of system-level properties (opacity, fluency, immediacy), mediated by attribution ambiguity and cognitive outsourcing."^kim-llm-fallacy-misattribution-2026
Manifestations in Education
| Domain | Educational Example |
|---|---|
| Computational | Student produces working code via Copilot but cannot explain logic, debug independently, or adapt to new requirements |
| Linguistic | Student generates fluent essay in a second language but cannot produce comparable prose unassisted |
| Analytical | Student presents structured step-by-step math solution but cannot replicate reasoning when AI is unavailable |
| Creative / Epistemic | Student reads AI summary of a topic and equates access to information with conceptual mastery (illusion of explanatory depth) |
| Professional signaling | Resumes, portfolios, and interview answers reflect ability to prompt LLMs rather than independently acquired expertise |
Relationship to Existing Wiki Concepts
- metacognition — The LLM fallacy is a metacognitive calibration failure: students cannot accurately monitor their own understanding because fluent AI output creates false fluency signals
- ai-learning-transfer — Misattribution undermines transfer because students believe they have mastered material they have merely prompted; the gap between perceived and actual capability manifests as transfer failure
- ai-tutor-safety-harms — SafeTutors' Cognitive (fluency illusion) and Ethical-Epistemic (misrepresentation) dimensions are tutoring-specific expressions of the fallacy
- self-regulated-learning — Attribution ambiguity disrupts the self-evaluation phase of SRL, preventing accurate causal attribution and adaptation
Implications for Education
Assessment integrity: Completed assignments may reflect LLM capability rather than student learning. Observable outputs are unreliable proxies when AI mediation is invisible (both to human and automated evaluators).
Metacognitive scaffolding needed: Interventions should help users correctly attribute contribution — e.g., requiring students to explain AI-generated content in their own words before submission, or using "explain-to-a-peer" protocols.
Tool design: Systems that surface their reasoning process (reducing opacity) or require iterative refinement by the user (reducing immediacy) may mitigate the fallacy.
Related Pages
- llm-cognitive-diagnosis-handwritten-math — MathCog benchmark: 18 LLMs evaluated on cognitive skill diagnosis from handwritten math; all F1 < 0.5; systematic over-attribution and hallucination of evidence (2025)
- contextual-sycophancy-ai-literacy — The Hidden Cost of Contextual Sycophancy: an AI Literacy Intervention in Human-AI Collaboration
- genai-meta-analysis-programming-learning — Productivity boost may mask underlying skill erosion from AI use
- brookings-ai-students-report — Brookings global report: AI risks overshadow benefits for students; 12 recommendations
- pedagogy-ai-mistakes — deliberately leveraging AI errors to foster higher-order thinking
- ai-learning-companions-framework — three-foundation framework for AI learning companions prioritizing durable learning over performance
- authentic-assessment — AI-authored work misattributed as student competence
- metacognition — How AI affects monitoring and self-evaluation
- ai-learning-transfer — Durability of perceived vs. actual gains
- ai-tutor-safety-harms — Pedagogical harms from fluent AI outputs
- self-regulated-learning — Self-evaluation and causal attribution in SRL
- educational-llm-alignment — Benchmark-based misaligned confidence as fallacy enabler
- educational-vlm-evaluation — Multimodal instances of misattribution in student work
- learning-by-chatting-genai-impact — Empirical evidence of misattribution of ChatGPT outputs to self-understanding
Sources
- Kim, H., Yu, H., & Yi, H. (2026). The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows. arXiv:2604.14807. PDF
- llm-reasoning-traces-metacognition — Reasoning traces as UI artifacts — not transparent windows into model cognition