Why AI Detection Fails for Academic Integrity

In a controlled study of 642 published English abstracts across four domains and two time periods (2013–15 vs. 2023–25), two commercial AI detectors at τ=0.50 flagged guideline-compliant light AI editing at 38–80%, flagged unmodified 2023–25 originals at 9–15% (non-STEM far above STEM, p<0.001), and after Undetectable AI humanization caught fewer than 4% of AI-labeled rewrites (FNR >96%). Detector scores therefore punished honest AI-assistance while enabling humanizer-assisted evasion — an "integrity catch-22" the authors argue should bar detector scores from serving as standalone misconduct evidence.

Relevance to AI in Education: This study supplies controlled, quantitative evidence for the conceptual critiques already in this knowledge base — Bassett et al.'s argument that detectors are unworkable, and the detection-reliability concerns running across Academic Integrity research. It uniquely quantifies the field-bias and humanizer-evasion problem, reinforcing the shift toward assessment redesign over surveillance.

Key Findings

• Compliant AI assistance is punished. "Refine (abstract only)" edits — a proxy for guideline-compliant, light AI assistance where the student retains intellectual ownership — were flagged at 38–49% by GPTZero and 64–80% by Pangram at τ=0.50, meaning honest AI editing carries substantial sanction risk.
• False positives rise in the LLM era, unevenly by field. On pre-LLM (2013–15) originals both detectors had 0% proxy false-positive rate, but on unmodified 2023–25 originals flag rates reached 8.9–14.9%; non-STEM domains (political science, theology) were flagged far more than STEM (chemistry, CS), p<0.001.
• Detection tracks style, not authorship. Elevated scores correlated with long-token ratio and Academic Word List (AWL) density (Spearman ρ≈0.30–0.35, p<0.001) rather than authorship intent — so "academic-sounding" prose, common among non-native and novice writers, raises false-positive risk independent of who wrote it.
• Humanizer evasion is near-total. After Undetectable AI humanization, fewer than 4% of AI-labeled rewrites remained flagged (FNR >96%); humanization systematically suppressed the same surface cues (long-token, AWL, lexical diversity) detectors key on.
• An integrity catch-22. The combination — high flags on honest assistance alongside near-total misses after humanization — means detector-based enforcement punishes acceptable use while failing to catch deliberate evasion. The authors recommend transparent AI-use norms and process evidence (drafting history) over standalone detector scores, and flag FERPA/GDPR concerns with uploading student work to third-party services.

Connected Concepts

• AI Detection
• Academic Integrity
• Equity In AI Education
• Language Learning
• Writing Education
• Assessment
• Educational Policy AI
• Generative AI
• Higher Ed

Connected Articles

• Bassett AI Detectors Education 2026 — Heads We Win, Tails You Lose: AI Detectors in Education
• Roe AI Humanizers Legitimacy Assessment 2026 — Dramaturgies of Deception: AI Humanizers and the Performance of Legitimacy in Higher Education Assessment
• Beyond Detection Authentic Assessment AI 2025 — Beyond Detection: Authentic Assessment
• Ssaho AI Academic Integrity Review 2025 — Review of AI-based plagiarism/AI-content detection reliability
• Detecting LLM Generated Text Latent Prompt — EchoPrompt: Detecting LLM-generated Text via Latent Prompt Restoration

Citation

Karr, J. A., Jr., Khvatskii, G., Hua, T., & Chawla, N. V. (2026). Why AI Detection Fails for Academic Integrity. Proceedings of the ACM AI Leadership Summit (AILS '26), Atlanta, GA.