Evaluating the accuracy and reliability of AI content detectors in academic contexts

Synthesis: Hadra, Cambridge and Mesbah tested the two commercial AI content detectors most widely deployed in higher education, Turnitin and Originality, against a balanced set of 192 texts: authentic EFL student coursework written before August 2022, professional human writing, AI-generated output from GPT-4.1 and Claude 3 Opus, and hybrid human–AI compositions mixed at roughly 50/50. Originality was the better performer overall (macro accuracy 0.69 against Turnitin's 0.61, macro recall 0.60 against 0.51), but both systems fell below a macro F1 of 0.55, both were close to useless on hybrid texts, and both lost accuracy significantly as texts grew longer and as the genre shifted from humanities to scientific writing. Originality also classified all 48 professional texts correctly while missing four EFL texts, a borderline-significant trend the authors read as a fairness risk for EFL writers. Their conclusion is policy-facing rather than technical: detector output can prompt inquiry but is not fit to be the sole basis for a misconduct decision, because misclassification of legitimate student work is the predictable cost of treating a probabilistic score as a finding.

Key Findings

1. Neither detector was accurate enough for the decisions it is used to support. Over the whole 192-text set, Originality reached macro-average accuracy of 0.69 against Turnitin's 0.61, and macro-average recall of 0.60 against 0.51. Both produced macro F1-scores below 0.55 (Turnitin 0.51, Originality 0.54), which the authors treat as the decisive number: consistency across the three authorship classes, not the headline accuracy, is what an integrity process would need.
2. Both systems behave like binary classifiers and fail on mixed authorship. Originality's sensitivity on Hybrid texts was 0.02 and Turnitin's 0.31, meaning that of the 48 hybrid texts, Originality correctly labelled essentially none. Specificity on that class was high (0.99 and 0.82), so the failure is one of omission rather than over-flagging: hybrid work is pushed into the neighbouring classes instead of being identified as mixed. The authors describe this as a structural limitation of tools calibrated for an AI-versus-human distinction.
3. Longer texts are classified less accurately. Turnitin scored 0.87 accuracy on 300–330 word texts, 0.56 on 450–550 words and 0.68 on 900–1,100 words; Originality scored 0.96, 0.63 and 0.84 respectively. Pearson's chi-square tests confirmed the association between text length and classification accuracy for both detectors, χ²(2) = 8.41, p = 0.0149 for Turnitin and χ²(2) = 13.17, p = 0.0014 for Originality. The pattern matters because the medium length band, where performance collapsed, contains 144 of the 192 texts and mirrors ordinary coursework.
4. Scientific writing is classified far worse than humanities writing. Turnitin's accuracy fell from 0.86 on humanities texts to 0.51 on science texts, and Originality's from 0.96 to 0.58, with both associations significant at p = 0.000 (χ²(1) = 19.109 and 26.429). The authors attribute this to features of scientific discourse, high lexical density, technical terminology and formulaic sentence structure, that resemble machine output. Their warning is directed at STEM faculties, where a large share of students could be systematically misread.
5. Originality showed a borderline bias against EFL student writing. It classified all 48 professionally authored texts correctly (100%) but misclassified four of 48 EFL texts (91.6%), a one-sided Fisher's exact test trend of p = 0.058. Turnitin showed no significant difference (93.8% professional against 91.6% EFL, p = 0.50). The authors do not claim proven bias, but argue the direction is consistent with earlier evidence that detectors penalise non-native writers, and that predictable lexical and syntactic patterns in developing writers may be read as machine authorship.
6. Detector percentages are vendor conventions, not measurements of authorship. The study mapped detector probabilities onto Human (0–20%), Hybrid (21–79%) and AI (80% or above) using thresholds that are standard in integrity investigations and published vendor guidance, not values derived from this dataset. Turnitin's own guidance no longer reports specific percentages below 20% because of false-positive risk. On this footing, a score in the AI band is a strong indicator of AI authorship, not proof that a student misused a tool.

How the study was conducted

The dataset was built deliberately balanced, with 48 texts in each of four categories, and the construction of each category is the reason the results carry weight for integrity practice. The EFL texts were genuine final coursework submissions from a university Foundation Program English for Sciences, all written before August 2022, when conversational LLMs were not yet in routine student hands; ethical approval was obtained and every submission was anonymised. The professional texts were sampled from the XSum corpus of BBC news articles, chosen to represent fluent, information-dense writing by experienced contributors. The AI texts were generated with GPT-4.1 and Claude 3 Opus from prompts built to reflect typical pre-undergraduate EFL essay tasks in scientific and humanities domains, then screened for coherence and genre fit. Hybrid texts combined roughly 50% EFL student writing with 50% AI output, edited for thematic continuity.

Length and genre varied naturally rather than equally: 23 texts were short, 144 medium and 25 long, and 57 texts were humanities against 135 science. The authors argue this unevenness is faithful to what students actually submit and that it does not compromise the analysis, since chi-square and Fisher's tests do not require equal group sizes. Detector outputs were dichotomised into correct and incorrect classifications against the ground truth, and accuracy, precision, recall, specificity, F1 and macro averages were computed from three-class confusion matrices. For the authorship comparison the subset was human-written throughout, so accuracy collapses into recall for the Human class and the analysis was restricted to that one metric.

Why hybrid writing breaks detection

The paper's central technical claim is that detector architecture has not kept up with how students write. The Hybrid class in this study is not an edge case: it represents the ordinary pattern of composing with AI assistance, and it is the class both systems fail. The authors note that institutions whose policies now permit limited or disclosed AI use are exposed on both sides of the same weakness, since a detector may miss legitimate declared assistance while simultaneously flagging authentic human work, and they conclude that as hybrid writing spreads, detection tools risk becoming less relevant to actual composition practice rather than more accurate.

Their reading of the length and genre effects reinforces the point. Detection methods rest on distributional features such as perplexity and lexical regularity, and those signals degrade exactly where texts get complex or where a discipline's conventions push human writing toward formula. Medium-length and long coursework, the writing that assessment actually depends on, is precisely where detector reliability is weakest. The paper places these findings against vendor marketing that portrays detection as reliable across contexts, and against the conflicting published evidence: Weber-Wulff et al. (2023) tested twelve public and two commercial tools and concluded they were neither accurate nor reliable; Hua and Yao (2024) found SVM-based approaches above 95% accuracy but GPT-Zero falling to 0.04–0.16 on paraphrased text; Chakraborty et al. (2024) argued detection remains possible if human and machine distributions stay distinguishable, while showing that the sample sizes required grow with generator quality.

Implications for institutional practice

The authors set out four directions rather than a detector fix. Institutions should integrate explicit instruction on responsible AI use; redesign assessment to focus on process, reflection and transparency rather than final product alone; develop policies that distinguish acceptable AI assistance from deceptive practice; and train faculty to interpret detector output cautiously and in context. Detection keeps a supplementary role inside a broader integrity workflow, but it is evidence that something deserves a closer look, not a verdict.

Two consequences follow for the specific population the study addresses. For institutions serving linguistically diverse students, the authors argue that excessive reliance on detectors reproduces linguistic disadvantage and makes the accuracy of authorship verification unequal across student groups, which is itself an assessment validity problem rather than only an equity one. They also name the cost of the opposite error: over-policing makes students feel monitored and damages Trust between students and institutions, while ignoring GenAI misuse compromises the validity of assessment. Both halves of that trade-off are treated as real, which is why the recommendation is human judgement, clearer policy and better-specified tasks rather than either surveillance or inaction. There is a parallel here with the argument that AI-generated text cannot by itself be equated with misconduct (Chan 2025), and with the paper's own framing that detection should be a prompt for inquiry rather than an authorship finding.

Limitations

The authors are explicit about four constraints. The 192-text dataset is balanced but modest beside large-scale benchmarking work, and a wider range of academic levels, genres and linguistic backgrounds would improve generalisability. Only two detectors were tested, so the results do not transfer automatically to other commercial, Open Source or emerging tools, whose behaviour also shifts with each model update. The hybrids were fixed at 50/50, whereas real students edit AI drafts to varying degrees or use assistance selectively at different stages, patterns the design could not capture. Finally, both generative and detection models continue to change quickly, so the authors call for longitudinal monitoring to establish whether detector reliability improves or fluctuates in ways that make consistent policy implementation impossible.

Connected Concepts

• AI Detection
• Academic Integrity
• Assessment Validity
• Reducing AI Misuse
• Generative AI
• Assessment
• Higher Education
• English Education (EAP / EFL / ESL)
• Language Learning
• Equity
• Bias Mitigation
• Trust

Connected Articles

• Heads We Win, Tails You Lose: AI Detectors in Education — The case against treating detector verdicts as evidence in education
• Why AI Detection Fails for Academic Integrity — Detection, humanisation and the arms race over authorship signals
• Detecting the Undetectable? Reassessing Academic Misconduct Procedures in the Era of Generative AI — What detection can and cannot show about misconduct
• Dramaturgies of Deception: AI Humanizers and the Performance of Legitimacy in Higher Education Assessment — Manipulating detector output to perform legitimacy
• Beyond Detection: How Students Use—and Hide—AI in Online Assessments and What Authentic Tasks Can Do About It — How online assessment design copes with AI authorship
• Generative AI and linguistic diversity in academic writing and publishing: Perspectives from World Englishes — Linguistic diversity and bias in AI-mediated academic writing
• Reimagining Success and Failure: Equitable Assessment Practices in an Age of Artificial Intelligence — Equity as a criterion for assessment design in the GenAI era
• The Integrity of Psychology Assessments in the AI Age: A Critical Examination — Programme-level evidence that the marking boundary, not detection, is the weak point

Citation

Hadra, M., Cambridge, K., & Mesbah, M. (2026). Evaluating the accuracy and reliability of AI content detectors in academic contexts. International Journal for Educational Integrity, 22(4).