Unchecked or Falsely Accused? Evaluating AI-Generated Text Detectors on Kazakh

Synthesis: Ozaman (2026) asks whether the AI-text detectors an institution can download today can be trusted on Kazakh, and finds they fail in both directions. The study releases AINA-KK, a Benchmark of 3,811 human passages whose existence before ChatGPT's release is provable, plus 5,597 machine and gray-zone passages from nine generators, including the Kazakh-built KazLLM and Sherkala. It documents unchecked accusations — TMR RoBERTa flags 81.4% of human passages and Desklib DeBERTa flags 79.6% of published student essays — and unchecked machine text, because the same detectors recalibrated to a 1% false-positive rate catch at most 1.3% of it. Most of these failures, the paper argues, have fixable causes rather than needing new data, and it packages the fixes as AINA-KK-Det, which flags 0.9% of human passages while catching 85.2% of machine text from known generators. For Academic Integrity practice the conclusion is blunt: no detector of this kind should stand alone as evidence of misconduct in Kazakh.

Key Findings

1. Five English-trained public detectors run as released accuse honest Kazakh writers: TMR RoBERTa flags 81.4% of human passages, Fakespot 25.5%, and Desklib DeBERTa 79.6% of published student essays.
2. Recalibrating those detectors to a 1% false-positive rate on Kazakh development text does not rescue them: none catches more than 1.3% of machine passages, while four still flag 1.6–2.5% of test humans.
3. Calibrated zero-shot detectors hold human error to 0.6–1.4% on unseen text but leave machine text unchecked; the best, Gemma-4-E2B Binoculars, catches only 50.3% of in-distribution machine passages.
4. Fine-tuned encoders rank machine from human text almost perfectly (AUROC 0.998), yet at the default threshold mDeBERTa flags 24.4% of 50-word human excerpts against 7.6% at full length.
5. Kazakh-built generators evade: calibrated mDeBERTa catches at least 96.8% of every global generator's passages but only 47.1% of KazLLM and 62.7% of Sherkala, and adding them to training is not a dependable fix.
6. Machine-transliterated Latin-2021 text defeats every calibrated threshold without script handling: 17 of 20 detectors catch at most 0.5% of it, and mDeBERTa's ranking inverts to AUROC 0.384.
7. AINA-KK-Det combines script canonicalization, script-augmented and length-matched training, length-specific thresholds and an OR with a zero-shot scorer to flag 0.9% of human passages and catch 85.2% of known-generator machine text.

How AINA-KK makes human authorship checkable

Before a detector can be fairly judged, there has to be human text whose authorship is not in dispute. AINA-KK supplies it: 3,811 passages from encyclopedic, news, web and educational pages that provably existed before ChatGPT's release, including student reports and essays drawn from 68 educational websites. Machine and gray-zone text adds 5,597 passages from nine language models — Gemini and Gemma models, gpt-oss-120b, Qwen3.8-27B, NLLB-200, and the Kazakh-built KazLLM and Sherkala — with script and length stress tests built in. Passages are clustered by topic and whole clusters are assigned to train or test, so no topic appears in both. The author, a native speaker, audited samples and released the judgments with the data: in the first pass 75.0% of human and 66.7% of machine items were valid, and a later sample judged only 33 of 52 human passages valid. That candor matters, because the reported false-positive rates describe public Kazakh text as found, not clean student writing.

Why short human texts are the sharpest failure

Length, not authorship, drives many false flags. Machine outputs in the benchmark are shorter than human passages — median 153 versus 175 words, with 18.5% against 0.5% falling under 120 words — so brevity itself becomes evidence of machine authorship. Cut human passages to their first 50 words and over-flagging spikes: mDeBERTa flags 24.4% at its default threshold, against 7.6% at full length. A threshold calibrated on full-length text brings the 50-word rate to 1.0%, but the script-augmented variant still flags 5.6%, and adding the two Kazakh-built generators to training pushes the 50-word false-positive rate to 82.2%. The paper's countermeasures are length-matched training, which weakens brevity as a cue, and length-specific thresholds applied to every input by its length. The practical consequence is that a detector's error rate is not one number: it depends on how much text a student actually submits.

Script, homoglyphs, and an alphabet in transition

Kazakh is written in Cyrillic today but is officially moving to a Latin-based alphabet, and real Kazakh text routinely mixes Latin look-alike letters into Cyrillic words: 5.2% of the benchmark's human passages contain them, 6.7% in the test split. Both conditions break English-trained tools, and they make Kazakh a hard case for multilingual detection generally. Transliterated into the 2021 Latin alphabet, mDeBERTa's ranking inverts (AUROC 0.384) and its detection of Latin machine text collapses to 0.0%; among the 20 detectors run without script handling, 17 catch at most 0.5% of it. Replacing 10% of eligible Cyrillic letters with Latin look-alikes cuts mDeBERTa's true-positive rate from 99.0% to 63.3%. Canonicalizing the script before scoring — mapping Latin-2021 back to Cyrillic and folding homoglyphs — lifts mDeBERTa to 99.0% on Latin machine text and 99.0% on homoglyph-attacked text without retraining, at a false-positive rate of 0.4% on canonicalized human text.

What a flag can and cannot mean

Gray-zone text is where tooling and policy part company. At its calibrated threshold, mDeBERTa flags 82.5% of passages that open with about fifty human words and are finished by a model, 7.7% of AI-polished human text and 7.9% of machine translations from Russian; the prompt-based humanizer is flagged 100.0% of the time. A flag therefore says little about how much of a passage a person wrote, which is a question of Assessment design and validity rather than detection. The authors also report precision at a hypothetical 5% prevalence: 82.9% of AINA-KK-Det's flags would be correct, against 6.9% of Fakespot's. Their ethics statement is unambiguous — no detector evaluated here, or known to them, should be used as sole evidence of misconduct in Kazakh — and their recommendation is that a flag is grounds for a conversation, never proof.

What this means for practice

• Instructors. Do not run English-trained public detectors on Kazakh student work: as released they flag 25.5% to 81.4% of human passages, and recalibrated they catch at most 1.3% of machine text.
• Instructors. Ask for the error rate at the length of text submitted rather than one headline figure, because 50-word excerpts are flagged far more often than full-length passages, 24.4% versus 7.6% for mDeBERTa.
• Instructors. Treat a flag as grounds for a conversation about the process, never as evidence, and tell students in advance that this is how the output will be handled.
• Administrators. Before licensing or advertising any Kazakh AI detector, require a false-positive rate measured on local human writing of the same genre and length, with an interval; a claimed "reduced to 1%" rate means little without one.
• Developers. Canonicalize script, fit thresholds by input length, match training-set lengths, and pair a fine-tuned classifier with a zero-shot scorer — fixes the paper shows need no new data collection.

Limitations

• Commercial detectors, including the local vendor whose "false positives reduced to 1%" claim motivates the work, were not evaluated, because they offer no free API.
• Human text is public pre-ChatGPT text as found, not essays collected from students; only 6 of 13 educational passages in the second audit were valid, so educational false-positive rates are the least certain.
• Both validity audits cover about 100 items, judged by a single annotator who saw each item's claimed class.
• The Latin-script test uses machine-transliterated Cyrillic with an exact inverse, not text written in Latin by people; informal Latin spellings and the 2017 and 2018 alphabets are not covered.
• Generators are limited to public API tiers and open weights, so the flagship models students most use are represented by lite or open siblings, and only 85 KazLLM and 75 Sherkala passages are test-split outputs.
• AINA-KK-Det was chosen with test results in view; on development data, where no configuration was trained, it flags 1.5% of human passages and catches 89.0% of in-distribution machine text.

Citation

Ozaman, M. (2026). Unchecked or falsely accused? Evaluating AI-generated text detectors on Kazakh. arXiv preprint arXiv:2610.15870.