Why AI Detection Fails for Academic Integrity

Synthesis: In a controlled study of 642 published English abstracts across four domains and two time periods (2013–15 vs. 2023–25), two commercial AI detectors at τ=0.50 flagged guideline-compliant light AI editing at 38–80%, flagged unmodified 2023–25 originals at 9–15% (non-STEM far above STEM, p<0.001), and after Undetectable AI humanization caught fewer than 4% of AI-labeled rewrites (FNR >96%). Detector scores therefore punished honest AI-assistance while enabling humanizer-assisted evasion — an "integrity catch-22" the authors argue should bar detector scores from serving as standalone misconduct evidence.

Relevance to AI in Education: This study supplies controlled, quantitative evidence for the conceptual critiques already in this knowledge base — Bassett et al.'s argument that detectors are unworkable, and the detection-reliability concerns running across Academic Integrity research. It uniquely quantifies the field-bias and humanizer-evasion problem, reinforcing the shift toward assessment redesign over surveillance.

Key Findings

• Compliant AI assistance is punished. "Refine (abstract only)" edits — a proxy for guideline-compliant, light AI assistance where the student retains intellectual ownership — were flagged at 38–49% by GPTZero and 64–80% by Pangram at τ=0.50, meaning honest AI editing carries substantial sanction risk.
• False positives rise in the LLM era, unevenly by field. On pre-LLM (2013–15) originals both detectors had 0% proxy false-positive rate, but on unmodified 2023–25 originals flag rates reached 8.9–14.9%; non-STEM domains (political science, theology) were flagged far more than STEM (chemistry, CS), p<0.001.
• Detection tracks style, not authorship. Elevated scores correlated with long-token ratio and Academic Word List (AWL) density (Spearman ρ≈0.30–0.35, p<0.001) rather than authorship intent — so "academic-sounding" prose, common among non-native and novice writers, raises false-positive risk independent of who wrote it.
• Humanizer evasion is near-total. After Undetectable AI humanization, fewer than 4% of AI-labeled rewrites remained flagged (FNR >96%); humanization systematically suppressed the same surface cues (long-token, AWL, lexical diversity) detectors key on.
• An integrity catch-22. The combination — high flags on honest assistance alongside near-total misses after humanization — means detector-based enforcement punishes acceptable use while failing to catch deliberate evasion. The authors recommend transparent AI-use norms and process evidence (drafting history) over standalone detector scores, and flag FERPA/GDPR concerns with uploading student work to third-party services.

What this means for practice

• Instructors. Stop treating a detector flag as misconduct evidence — 38–80% of guideline-compliant light edits were flagged — and instead pair any flag with drafting history, reserving sanctions for cases where human review of the student's process supports a lack of effort.
• Instructors. Require process evidence such as drafts, version history, and revision notes, so integrity judgments rest on how a paper developed rather than on the stylistic surface features that raise detector scores.
• Administrators. Audit any detector policy for disparate impact before adopting it: non-STEM prose (political science, theology) was flagged far more than STEM text, and academic-sounding writing raises flag risk for non-native and novice writers irrespective of authorship, an Equity concern.
• Policymakers. Regulate disclosure rather than ban AI use. Humanizers suppressed detection to under 4% of flagged rewrites (FNR >96%), so an enforcement regime built on detector scores punishes honest assistance while missing deliberate evasion.
• Administrators. Complete FERPA and GDPR review before uploading student work to third-party detection or humanization services.

Limitations

• The corpus was 642 published English abstracts across four OpenAlex domains, not student essays, so detector behavior on shorter, classroom-shaped student writing may differ.
• Ground truth was proxy-based — unmodified abstracts treated as human and LLM rewrites as AI — so 2023–25 positives are flag rates rather than confirmed false positives, all measured at a fixed threshold of τ = 0.50.
• One rewrite generator (Gemini 3 Flash), two detectors (Pangram 3.2, GPTZero), and one humanizer configuration (Undetectable AI v11) were used; results may not generalize to other models, versions, humanizers, or prompting strategies.
• The study covers English only and should not be extrapolated to multilingual classrooms, and corpus size was constrained by API and humanization cost — 800 abstracts were targeted and 642 retained after filtering.

Citation

Karr, J. A., Jr., Khvatskii, G., Hua, T., & Chawla, N. V. (2026). Why AI Detection Fails for Academic Integrity. Proceedings of the ACM AI Leadership Summit (AILS '26), Atlanta, GA.