Has the Target Moved Again? GenAI and the Validity of English Proficiency Scores

Synthesis: This commentary argues that high-stakes English proficiency tests — TOEFL iBT, IELTS, PTE Academic, and the Duolingo English Test — still require unaided, timed reading and writing, while students now routinely use GenAI for the same tasks in academic life, and that this gap is a validity problem rather than only a design issue. Drawing on Kane's argument-based approach, the author locates the difficulty in the extrapolation inference, the step from unaided scores to real academic performance, because test tasks and conditions no longer resemble the target domain for writing and reading. A review of recent studies suggests AI supports writing and reading most, listening somewhat less, and speaking least, with uneven evidence across the four skills. After considering two strong objections, the author offers three tentative directions: reframe unaided proficiency as a foundation for supervising AI output, treat AI-mediated performance as a separate construct, and investigate the question through predictive validity research. The piece is explicitly early brainstorming and closes with its own limitations.

Key Findings

1. High-stakes English proficiency tests require unaided, timed reading and writing, but many students now use GenAI for those same tasks in their academic study.
2. In Kane's argument-based framework, the extrapolation inference — from unaided scores to real academic performance — is the step most affected, not the scoring or generalization steps.
3. Writing has the strongest evidence: cited surveys report over 80% of 634 US college students using GenAI academically, and 88% of 1,041 UK undergraduates using it for assessments, up from 53%.
4. AI assistance appears to help lower-proficiency writers most: ChatGPT-assisted revisions raised scores most in vocabulary and grammar in one controlled study, and a randomized experiment with professional writers found the tool narrowed the gap between weaker and stronger writers.
5. Reading evidence is smaller and mixed: AI tools improved comprehension for lower-performing readers but significantly worsened it for higher-performing readers, who leaned on summaries and lost depth (Etkin et al., 2025).
6. For listening, live transcripts did not improve final grades overall but significantly increased class participation for lower-proficiency learners (Wang & Chen, 2023); for live academic speaking the author found no peer-reviewed study of AI mediation.
7. The commentary proposes three tentative responses: reframe unaided proficiency as a supervisory or foundational literacy, explore AI-mediated performance as a separate construct, and test the concern through predictive validity research.

The gap between test tasks and academic work

For instructors, the question is blunt: when a proficiency score lands on an admissions record, what is it a claim about? The new TOEFL iBT, which started in January 2026, added a writing task called Write an Email but kept its Write for an Academic Discussion task; IELTS Academic has two writing tasks (describing visual information, and writing an argumentative essay); Pearson's PTE Academic has a Summarize Written Text task and a full Write Essay task; the Duolingo English Test has Write about the Photo and Write about the Topic. Reading sections follow the same logic, using long academic passages with inference and vocabulary-in-context questions.

The through-line is unaided, timed production and comprehension of text types made as authentic as possible. That framing leans on Bachman and Palmer (1996), who defined authenticity as how closely test task features match a real task outside the test and distinguished it from interactiveness. Lewkowicz (2000), though, found that of 72 university students who took both a simple multiple-choice test and a more realistic one, only 12.5% preferred the more realistic test because it felt like real academic work. Test takers may care about authenticity less than designers assume.

Why the extrapolation inference is the vulnerable step

The commentary's central move is to frame the problem in measurement terms. Validity concerns the interpretations of scores for proposed uses, and it is the interpretations — not the test itself — that are evaluated (AERA et al., 2014). The same standards name two ways a test can misrepresent its construct: construct underrepresentation, when the test misses something the construct needs, and construct-irrelevant variance, when scores reflect something outside the construct. Messick (1989) adds that a score's meaning depends on whether conclusions drawn from it stay accurate over time.

Kane (2013) splits the path from performance to claim into scoring, generalization, extrapolation, and — for decisions — a decision inference, each needing its own evidence. Extrapolation, he notes, is especially open to challenge when test tasks and conditions differ meaningfully from the target domain. So if real-world writing and reading are now shaped by tools the test forbids, extrapolation is the step needing scrutiny. If tests reward only unaided performance, preparation courses may keep training skills students use less and less while neglecting the ones they increasingly need, such as checking and correcting AI output — a form of negative washback.

Two objections, and why they are not quite enough

The author states the strongest counterarguments rather than burying them. The first is that these are proctored exams — test centers and Remote Proctoring software exist precisely to stop outside tool use — so the test never claimed to predict AI-assisted performance. He accepts the point but argues it is more than a communication problem: universities, employers, and immigration authorities rarely explain that distinction when they act on a score, and research on score users finds many admissions staff read scores in broad, general terms.

The second objection is stronger: a student still needs real command of English to use AI well — to write a clear prompt, to judge whether output sounds right, to notice when a summary misses the meaning. The author largely agrees and names the relevant capacity supervisory proficiency: the language knowledge needed to direct a tool, evaluate its output, and correct or reject it, which overlaps with AI literacy but centers on language rather than technical skill. His residual worry: a test built only around unaided reading under time pressure does not reveal which candidates know when to trust an AI summary — and independent reading proficiency predicted which readers were helped and which were harmed.

Redefining the construct, and what redesign does not fix

The commentary closes with three tentative directions, offered explicitly as brainstorming. The first keeps unaided proficiency as the construct but changes its justification, reframing it as a supervisory or foundational literacy and being more honest in score reports about which claim a score supports. The second borrows Godwin-Jones's (2024) term distributed agency and asks whether something like it could be measured as its own construct alongside, not instead of, unaided proficiency — while acknowledging this stretches well beyond what he argues, since he builds the idea to describe and teach rather than to test. Two problems follow: rewarding AI-tool skill could end up measuring prior AI exposure and access as much as English proficiency, and if tools entered the test condition, providers would have to fix which tool, version, and settings — GenAI tools update often and can vary in output for the same prompt, which could weaken the generalization inference. The third direction treats the matter as an open empirical question: follow newly admitted students, collect their unaided scores, and have subject lecturers rate both their AI-assisted coursework and comparable work done without AI, testing whether unaided scores still predict well.

The author also places this in existing views of construct definition via Chapelle (1998), whose interactionalist approach includes both the learner's ability and the context in which it is used — and the tools available in the target language use domain are part of that context. Recent test changes do not resolve the gap: TOEFL's 2026 redesign changed delivery, timing, and scoring, with writing and speaking classified as AI-scored task types. The author separates AI used by the provider to score responses from AI used by test takers to produce language; the redesign changes the first, the commentary the second. Drawing on Fulcher and Davidson (2009), he distinguishes an upgrade retrofit — making a test serve its original purpose better — from a change retrofit, which extends it toward a new purpose or claim, and notes that extending a test's purpose without a new interpretive argument is, in their terms, not good practice.

What this means for practice

• Instructors. Treat a strong unaided score as a baseline for directing and checking AI output, not proof of independent writing in AI-rich coursework; ask students to show how they evaluated and revised AI-generated text.
• Instructors. Teach the supervisory skills the tests do not measure — deciding when to trust an AI summary, detecting when a tool has quietly changed the intended meaning, and correcting or rejecting output when it is wrong.
• Administrators. Say explicitly which claim a proficiency score supports — unaided performance under exam conditions or AI-assisted academic performance — in admissions and placement communications, since research on score users finds many read scores in broad terms.
• Researchers. Run the predictive validity study the commentary proposes: follow newly admitted students, record their unaided test scores, and have subject lecturers rate both AI-assisted coursework and comparable unaided work with AI-use conditions clearly documented.
• Developers. If AI-mediated tasks are added to a test, settle the standardization questions first — which tool, which version, which settings — or risk weakening the generalization inference and making scores hard to compare across test dates.

Limitations

• The piece is a commentary, described by the author as "an early piece of brainstorming, not a finished argument"; it reports no study of its own and no effect sizes.
• The evidence is uneven across the four skills — writing is most developed, reading and listening rest on fewer studies, and the speaking claim is an absence of evidence — so the commentary is more confident about writing than the rest.
• Several supporting figures come from surveys, one qualitative study (Dai, 2026, interviewing 12 postgraduate L2 students in UK higher education), and the author's own informal teaching observation that student writing now arrives polished — not from controlled research on test performance.
• The argument may not apply equally across contexts: institutions and test-taking groups differ a lot in AI access and policy, so what is true in a well-resourced context may not be true elsewhere.
• Testing the concern cleanly is hard because the baseline is already noisy: correlations between proficiency scores and first-year grades are usually small, sometimes not significant, and vary by subject area, degree level, and nationality (e.g., Isaacs et al., 2023).

Citation

Gönülal, T. (2026). Has the Target Moved Again? GenAI and the Validity of English Proficiency Scores. EdArXiv preprint.