Detecting GPT-Assisted Writing Using Interpretable Stylometric Features

Synthesis: This study asks whether nine interpretable stylometric features, computed from submitted text alone, can support detection of ChatGPT-assisted student writing. The data are 180 samples from 90 participants who wrote the same prompts twice, once unaided and once paraphrasing GPT-generated content. Eight classifiers were trained on 72 participants and tested on 18 unseen ones, using 250-word windows and a median-probability rule for document-level calls. Random Forest reached an ROC-AUC of 0.870 and an F1-score of 0.842 on 36 held-out documents, but flagged 4 of 18 independently authored documents as GPT-assisted. SHAP attribution puts the weight on lexical features, especially hapax ratio and non-stopword ratio. The authors present this as decision support for Academic Integrity review, not proof of misconduct.

Key Findings

Interpretable features carry real signal. Random Forest reached an ROC-AUC of 0.870 (95% CI 0.747–0.981) and F1 0.842 on 36 held-out documents from 18 unseen participants.
The false positives are the headline risk. Four of 18 independently authored documents were flagged as GPT-assisted (22.2%, 95% CI 9.0–45.2%) and two GPT-assisted documents were missed (11.1%, 95% CI 3.1–32.8%).
Lexical features did most of the work. Feature-group ablations showed lexical features carried most of the discriminative signal, then part-of-speech features, while Sentence-Length Variability alone performed near chance.
GPT-assisted text was more lexically diverse. It scored higher on type-token ratio (0.65 vs. 0.60), hapax ratio (0.51 vs. 0.44), word entropy (6.94 vs. 6.74) and non-stopword ratio (0.64 vs. 0.59).
Disjoint users and fixed windows guard against inflated scores. Training drew on 72 participants (144 samples, 595 windows), testing on 18 (36 documents, 127 windows), in 250-word windows with a 125-word stride.
SHAP ranked hapax ratio highest. Hapax Ratio was most influential, followed by non-stopword ratio, noun ratio, adverb ratio and type-token ratio; adjective ratio contributed least.
Document length explains part of the pattern. Independently authored responses averaged 620 words (SD = 198) against 515 (SD = 155), and a word-count-only classifier reached ROC-AUC 0.68 under cross-validation.

What the nine features measure

The detector rests on nine window-level measures: type-token ratio, hapax legomena ratio, word entropy, non-stopword ratio, noun, verb, adjective and adverb ratios, and sentence-length variability, computed through a text-analysis pipeline of quanteda, udpipe's English EWT part-of-speech model, and a 175-word Snowball stop list. Across the 90 participants, GPT-assisted responses were lexically richer and more noun-heavy: type-token ratio 0.65 against 0.60, hapax ratio 0.51 against 0.44, word entropy 6.94 against 6.74, noun ratio 0.27 against 0.24. Independently authored text ran higher on adverb ratio (0.05 against 0.04) and sentence-length variability (14.67 against 13.07).

Windows, disjoint users, and the leakage problem

Because participants wrote under both conditions, a random split could put one person's texts on opposite sides and leak author style into evaluation. Working from the IIITD-BU (Paraphrased) dataset of university students in a course on large language models, they kept every window from a participant in one fold and held 18 people out entirely. Documents became 250-word windows with a 125-word stride and 50% overlap, yielding 722 windows, 508 of them full 250-word windows. Window probabilities were combined by median and thresholded at 0.50. Tuning used 30 Bayesian optimization iterations on training data only; among eight classifiers Random Forest ranked first at validation ROC-AUC 0.838, k-Nearest Neighbors last at 0.763.

How the model fails, and what the SHAP values explain

The median rule misclassified six of the 36 held-out documents: four false positives with medians from 0.636 to 0.920, and two false negatives at 0.495 and 0.151. Across train-test ratios from 60/40 to 80/20 the mean FPR moved to 12.4%–15.8% and the mean FNR to 23.7%–24.8%, which the authors read as evidence that one small test set carries substantial uncertainty. Wilson and bootstrap intervals keep the uncertainty visible without making a 36-document estimate valid for high-stakes decisions. SHAP importance put hapax ratio first, then non-stopword ratio, noun ratio, adverb ratio and type-token ratio. A correctly flagged window scored 0.947 against 0.077 for an independently authored one, and a local explanation covers one window, never the document-level aggregation.

What a detection score can and cannot decide

A 22.2% false positive rate is a problem about people, not metrics: each false positive is independently authored work accused of GPT assistance, a decision that can affect grades and disciplinary action. The authors therefore frame the model as decision support that should trigger contextual review rather than replace instructor assessment, student testimony, or due process, which puts human review at the center of any workflow. They report no evaluation across demographic or language-background groups and so draw no fairness conclusions, studied only the paraphrase form of assistance, and call for external validation before institutional use. Against 57% of U.S. college students reporting at least weekly AI use in coursework, the paper states its limitations as plainly as its results.

What this means for practice

Instructors. Treat a GPT-assisted prediction as a reason to ask about process, not as evidence: 4 of 18 independently authored documents were flagged here, so request drafts or a conversation first.
Administrators. Do not adopt stylometric detection as an automated misconduct screen; a 22.2% false positive rate with a 95% CI of 9.0%–45.2% rules out standalone use, so fund review capacity instead.
Researchers. Replicate on other institutions, tasks, languages and student populations and report subgroup error rates, since this study estimated none for demographic or language-background groups.

Limitations

The sample is 90 participants from one course on large language models at one institution in India, writing on one topic under one assistance setting: paraphrasing GPT output.
The headline estimate rests on 36 documents from 18 participants, so intervals are wide (FPR 9.0%–45.2%; FNR 3.1%–32.8%) and error rates shifted under alternative train-test splits (mean FNR 23.7%–24.8%).
Writing condition is confounded with session order, because every participant completed the independent session first.
Other forms of AI assistance — brainstorming, editing, or generating text without paraphrase — were not evaluated.

Connected Concepts

Connected Articles

Heads We Win, Tails You Lose: AI Detectors in Education — Heads We Win, Tails You Lose: AI Detectors in Education
Why AI Detection Fails for Academic Integrity — Why AI Detection Fails for Academic Integrity
Who wrote this? Evaluating the reliability of AI detection tools in higher education — Who wrote this? Evaluating the reliability of AI detection tools in higher education
Evaluating the accuracy and reliability of AI content detectors in academic contexts — Evaluating the accuracy and reliability of AI content detectors in academic contexts
Detecting the Undetectable? Reassessing Academic Misconduct Procedures in the Era of Generative AI — Detecting the Undetectable? Reassessing Academic Misconduct Procedures in the Era of Generative AI
Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration — Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration
Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content — Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content
Beyond Detection: How Students Use—and Hide—AI in Online Assessments and What Authentic Tasks Can Do About It — Beyond Detection: How Students Use—and Hide—AI in Online Assessments and What Authentic Tasks Can Do About It

Citation

Kumar, R., Siddiqui, N., & Fuchsberger, A. (2026). Detecting GPT-Assisted Writing Using Interpretable Stylometric Features. arXiv:2609.26687.

Connected FAQs

Should We Use AI Detectors?

Metadata

Study design	Secondary Analysis
Discipline	Writing Education
Level	Higher Ed
Intended audience	Instructors, Researchers
Technologies and techniques	Machine Learning, Large Language Models (LLMs), Educational NLP
Assessment and measurement	AI Detection, Assessment Validity, Educational Measurement
Research methods and evaluation	Quantitative Research, Research Methods in AIED
Equity, ethics, and responsible use	Explainable AI, Bias Mitigation, Differential Effects Across Learner Groups
Foundations of AI in education	Academic Integrity, Reducing AI Misuse, Limitations in AIEd Research
Page kind	Evaluation

Source History Edit Chat