AI in Education Knowledge Base — AIEd Research Weekly Update

AIEd Research Weekly Update #1

Sat, 24 Oct 2026 08:00:00 -0400

A weekly brief from the AI in Education Knowledge Base, on what the research published in the past week means for instructors, developers, researchers, and administrators.

This week brought 68 new studies, taking the knowledge base to 1,802 article summaries, 220 concept pages, 37 FAQs, and 31 resources. Search or use AI to chat with the knowledge base.

Main highlights and questions
For instructors
For developers
For researchers
For administrators
Concept spotlight
Also added this week
Site notes

Main highlights and questions

Both of this week's headline results say the same thing: a score is not the outcome. A three-arm trial lifted exercise scores from 29.85% with the open web to 71.84% with unrestricted ChatGPT while conceptual learning did not differ at all, so a higher mark under AI access is not evidence that students learned more. Thirteen detectors flagged human-written manuscripts at rates from 0% to 100% after a single copy-edit, so a detector flag is not evidence of misconduct.

Does the tutor that lifts the homework also lift the exam? A randomized trial put a proof tutor in front of students and then measured an exam taken without it.
When a submission looks fine, how much of it is the model's? Fifty-nine chatlogs, compared against the work the same students handed in.
How often does an AI detector flag writing a human actually wrote? Thirteen detectors, 135,389 document pairs where only the language changed.
Is AI speaking practice improving speaking, or one measured feature of it? Ten studies, six CEFR features, and four studies that reported a single holistic score.
Does teaching students to use feedback build their engagement, or the other way round? A three-wave panel that cannot order the two.
Where do AI tutors still fail when a student refuses, withdraws, or escalates? 923 dialogues built from case studies of real students, scored against what practitioners documented.
Is AI governance in education law, guidance, or aspiration? 160 documents, of which four take binding form.

For instructors

A tutor can lift the homework and leave the exam where it was

Chen and colleagues randomized 153 proof-writing students at Carnegie Mellon to get a proof tutor early or late, then measured an exam taken with no AI in the room. Homework scores rose 2.56 points while the tool was available. Midterm scores did not differ. Once everyone had access and chose their own dose, every 100 additional chatbot messages went with a 3.02-point lower score on the next exam, and students with lower baseline self-efficacy were the ones who used it most.

What that changes for you: a grade earned with the tutor open is not evidence of what a student can do without it, so the unassisted task has to carry that weight. It also means the students most at risk of leaning on it are the ones least confident going in, which is visible in usage before it is visible in scores.

Unrestricted chat inflates scores more than a well-built tutor does

In a three-arm trial at a Munich CS1 course, 275 students spent 90 minutes on a Java exercise with a hint-giving tutor, with unrestricted ChatGPT, or with the open web. Exercise scores came out at 71.84%, 57.50% and 29.85%. Conceptual knowledge gains were 0.83, 0.71 and 0.85 points, and the difference between conditions was nowhere near significant (p = .773). Both AI conditions reduced frustration; only the tutor raised motivation.

The score spread is mostly about the artifact, not the learning. If you grade the exercise, the AI conditions are being graded on a different task.

Teacher-built chatbots fail at intent, not facts

EduWild collected 1,726 conversations from 217 teacher-configured Playlab bots, including 900 from the students they were deployed to. Bots skipped required workflow steps in 53.8% of end-user conversations, never established their role in 33.4%, and overloaded learners with information or questions in 37.1%.

Before you deploy a bot you built or inherited, read its first turn and check that it states what it is and what happens next. The faults documented here are in the instructions, and they are fixable in the prompt without touching the model.

A detector verdict is not evidence

An editing-service dataset gave Park and colleagues 135,389 document pairs that hold author, topic, and content fixed while only the language changes. Across 13 detectors, false positives on human-written academic text ran from 0% to 100%, and the same edits moved detectors in opposite directions. Separately, a group of sessional markers lost their detector entirely because it flagged grammar checkers, and now read for fabricated references and disconnection from course materials instead.

If you have been putting a detector score in front of a student, especially a student writing in a second language, this is the week to stop using detectors. The reading-based signals in the second study are the practical alternative.

Assessment design is a choice among three constraints, not a search for a principle

Pedlow, Fischer, and Williams argue the problem is not a shortage of design principles but the absence of a way to reason about the ones that pull against each other. Their dual trilemma treats authenticity, observability, and feasibility in assessment, and specificity, timeliness, and workload in feedback, as constraints you trade off deliberately.

Asked to redesign an assessment for AI, that framing turns an open question into a short argument you can record and defend.

Speaking gains from chatbots are uneven by feature

Soriano-Gonzalez and Belda-Medina screened 306 records down to ten studies and asked which of the six CEFR features of spoken language actually moved. Range and intelligibility improved in four studies each, coherence in one, interaction in almost none. Four of the ten reported a single holistic score, so their gains cannot be traced to a feature at all.

A vendor claim of "improved speaking" is compatible with a change the study never measured. For listening, Goh and Aryadoust find the evidence much thinner.

Feedback literacy and engagement feed each other

A three-wave panel of 399 undergraduates found AI feedback literacy predicting later agentic engagement (β = 0.222) and the reverse path just as strong (β = 0.263), with no statistical basis for ordering the two.

Teaching students to read and use feedback is not a prerequisite you finish before engagement starts. The two move together, so a course that only builds one of them is working against itself.

For developers

Hard cases are where tutors still fail, and strategy retrieval helps mid-tier models

TutorCaseBench turned qualitative case studies into 923 six-turn dialogues across five challenge types and scored 11 models against case-specific rubrics. The best model satisfied a documented intervention in 55.8% of open-ended cases, which is the number to design against. Supplying the case literature's documented strategy lifted 10 of 11 models by an average of 9.9 points, with Llama 4 Maverick moving from 41.0% to 64.8%, while the strongest model stayed flat at 71.3%.

Retrieval of reference strategies is worth building for the models you can afford. It will not rescue a frontier model that already has the pattern.

Validate against deployed conversations, not lab prompts

The EduWild dataset pairs every conversation with the system prompt and settings behind it, which is what makes the failure analyzable: 53.8% of end-user conversations skipped a required step. Teacher-authored configuration is a distinct failure surface from model capability, and a prompt-level harness recovered intent without removing the teacher's control.

If your product lets teachers write the instructions, your test set has to be their instructions, collected after deployment.

The rubric dimension of feedback quality is the one nobody measures

Yang examined 68 studies of AI-generated feedback. Evidence validity appears in 47 of them, pedagogical usefulness in 58, and rubric and task alignment in 19. Alignment is the dimension a product can most directly control, and it is the least studied.

That is an opening: build against the rubric the assignment already uses, and say so, because the published evidence will not tell buyers whether anyone else does.

Actionability is a design target, not a nicety

Tamme, Schilling, and Hantel propose replacing the end-of-semester written survey with a phone interview run by an AI agent, on the argument that the written instrument fails on response rate, depth, and actionability. It is a proposal rather than a field test. The design claims are specific enough to borrow: ask follow-ups, require a reason, return something the recipient can act on.

For researchers

Self-report and performance often reach different conclusions

Fan and colleagues mapped measurement methods across 2,099 records, then examined 51 cases where a self-report instrument and a test, rating, or trace measured the same construct. The answer to whether the two agree is often no. Where they diverge, the construct's observability explains it better than study quality does.

Substituting one for the other is a measurement decision with a direction, not a convenience. Report which one your claim rests on.

Artifacts hide offloading, so measure the process

Kontorovich and Wallach coded the 429 prompts in 59 chatlogs as self-regulation traces and compared each against the submitted work. Ten of the fifteen students who completed all three tasks changed which role they gave the model as the task changed, yet 53 of 59 submissions reproduced the model's product or revised it only in form.

A submission alone cannot distinguish use from substitution. Process data can, and the coding scheme here is reusable.

A validated scale for reliance in open-ended tasks

Most reliance measures infer dependence from whether a student followed correct advice, which needs a checkable answer. Hou and colleagues built and validated a 16-item scale for problem-based learning, where no such answer exists.

Detector benchmarks decay, so report the adversary

Ren, Raghavan, and Garg separate the detection question into separability and motion. Supervised baselines separate human from machine prose almost perfectly in distribution, while the commercial detector Pangram fell to 63.5% balanced accuracy and 27% recall against a prompt written to evade it.

Numbers reported without the evasion attempt describe a world that no longer exists. And the review literature is thin where it matters: of 51 studies of GenAI feedback on second language writing, 5 check whether the feedback is accurate.

For administrators

Most education AI governance is guidance, not law

Elmahjub and colleagues read 160 English-language governance documents closely, from 498 records. Only 4 take formal legal or regulatory form, while 92 are guidance, frameworks, toolkits, or handbooks. Accountability appears in 146 documents but is paired with a named oversight actor in 100.

A policy without a named owner and a mechanism is a statement of intent. The review gives you the fields to audit your own document against.

Detector procurement needs a false-positive number

Thirteen detectors, human-written academic text as the control, false positives from 0% to 100%. At least one institution has already retired its detector because it flagged grammar checkers. If a vendor will not give you a false-positive rate on human work in your own discipline, you cannot price the disciplinary risk.

The documented direction in assessment is process, authenticity, and oral work

Crompton and Burke review 72 studies from 24 countries and 20 disciplines and find three design responses recurring: process-oriented, authentic, and oral assessment. The evidence base is wide and methodologically thin, most studies cover a single course, and none of that changes the direction.

Adoption has outrun understanding among teachers

Among 132 Indonesian K-12 teachers, 96.2% support teacher use of AI and 84.6% support student use, yet 69.2% gave a vague or mistaken account of what a chatbot does with an instruction.

Support in a survey is not capability in a classroom. The professional development gap here shows up before any tool deployment does, and the instrument is short enough to reuse as a needs assessment.

Replace the course survey only if the replacement is actionable

The voice interview proposal targets the failure modes that make end-of-term surveys useless: response rate, depth, satisficing, and answers nobody can act on. Treat it as a specification for whatever you use instead, not as evidence that the AI version works.

Concept spotlight

Four concept pages carried this week's findings into the synthesis layer. Each is a standing page rather than a one-off, so the new evidence sits alongside what the page already argued.

Language learning

The page now says plainly that evidence on chatbot speaking gains is uneven by feature, and that four of the ten synthesised studies reported one holistic score, which is why their gains cannot be attributed to a feature. Read it before buying or building oral practice.

Intelligent tutoring

Its new finding is the retrieval result: supplying reference strategies lifts weak tutors and leaves strong ones where they were. That is a build decision, and the page separates it from the older claims about hint quality.

Self-report measures

The sharpest new case of the self-report gap is intention, where a single score covered learning-oriented, shortcut-oriented, and mixed use in the same sample of 340 students, with 68 re-administered items and eight interviews behind the claim.

Governance

The page now leads with the instrument-form finding: 4 of 160 documents binding, 92 advisory, and accountability named as an actor in only 100. It reframes governance questions from what a policy says to whether it can act.

Also added this week

The other studies, with methods and abstracts, are listed by day on the journal page.

Site notes

Translations

Seventeen locales received content updates this week: Spanish across 248 files, fourteen more between 38 and 47 files each, and single pages in Vietnamese and Bengali. All 32 locales also received the rewritten home, AI-assistant, and resources copy. 404 copy was added for eight more locales, and the 404 page now renders in the locale its path names.

Copy

The home page, the AI-assistant page, and the resources page were tightened: no em dashes, plainer leads, and side explanations that had been narrating the page removed. The copyable prompt now points at the reader's own locale's pages and names the real sizes of the offline files.

Tooling

The locale-module checker was rewritten to read every string shape in a copy module, after it turned out to be skipping most of the text it was meant to check. AI model attribution is gated against the registry rather than a list of allowed names. The concept screen widened to discipline and level labels. 186 wikilinks that repeated their own page title were fixed. The offline editions are served from the repository instead of the site, and Astro moved to 7.4.

What was edited

126 of the concept pages were rewritten this week, all 37 FAQs were touched (the body heading that repeated each page title is gone, and each page records its audience), and 11 resource entries were edited.

AIEd Research Weekly Update comes from the AI in Education Knowledge Base, which an AI agent generates. This issue was drafted by that same AI agent, with editorial guidance from the site developer, and every claim above links to the page it came from.

The knowledge base is dedicated to the public domain under a CC0 1.0 license. No rights reserved.

Found an issue with the content? Report it on GitHub, or contact the site developer, Doug Holton.

AIEd Research Weekly Update #0

Fri, 23 Oct 2026 20:46:45 -0400

A weekly brief from the AI in Education Knowledge Base, on what new research in AI and education means for instructors, developers, researchers, and administrators. This first issue introduces the site itself, because a link to a page is only useful once you know what kind of page you have landed on.

It holds 1,802 article summaries, 220 concept pages, 37 FAQs, and 31 resources, in 33 languages. Search or use AI to chat with the knowledge base.

Why this site exists
How it is organized
Where to start, by role
Questions the site already answers
Search it or chat with it
Read the whole thing offline
Read the site in your language
What comes next

Why this site exists

AI in education research arrives faster than anyone can read it: thousands of papers a year, most of them reporting one course, in one institution, under conditions the abstract does not state. An instructor who wants to know whether AI feedback improves writing has to find the studies that bear on that question, read them against each other, and notice that one of them tested a model two generations old. That is research work, and nobody hired that instructor to do it.

The site does that reading in public. An AI agent reads the open-access literature and drafts the pages, a human editor reviews and publishes them, and every claim ends in a link to the paper behind it, so a sentence you doubt can be checked in two clicks. It is free, it is in the public domain under CC0 1.0, and there is nothing to sign up for.

It is written for the people who teach, study, design, or administer AI in education, which is why one finding often appears three times in different places: once for the instructor who has to grade with it, once for the developer who has to build it, and once for the administrator who has to decide whether to buy it.

How it is organized

Four kinds of page, and the kind tells you what it is good for:

Concept pages collect what the research says about one idea, such as feedback, assessment, or cognitive offloading, and link every study behind the claim. Start here for the overview.
Article summaries cover one paper at a time, with its findings, its method, its limitations, and the full citation. They are listed day by day on the journal page.
FAQs answer the practical question in the words people actually ask it, with the research attached.
Resources list tools, collections, and instruments worth using, with who made each one, what it costs, and when its link was last checked.

The left sidebar indexes everything by strand, and the concept map on the home page arranges the field into twelve strands with AI in Education at the center. Every page ends with what it connects to, so a topic can be followed across the whole set: from a concept to the papers behind it, from a paper to the concepts it touches, and from an FAQ to both.

Where to start, by role

Questions the site already answers

Most of the questions people put to a chatbot have already been answered here by hand, with the research attached. A few worth opening before you ask:

Should we use AI detectors? Not as evidence in a misconduct case: a detector score cannot be validated against ground truth, and its errors are patterned rather than random.
Does using AI actually help my students learn? It can, but better work produced with AI is not evidence of better learning.
How do I redesign assessment so that a grade still tells me something defensible? Name what the grade is meant to represent before choosing the task.
How do I keep students from over-relying on AI? Graded work that reads better than the student can explain is a design problem, not a policing problem.
How do I write a course AI policy and communicate it to students? It has to be fair to the student who plays by the rules and the one who does not, and something you can apply to forty submissions.
How do I teach students to verify AI output? Warnings to check the sources ask for a disposition; a checklist and a worked example do not.
How can AI support disabled and neurodivergent learners in my course? What to adopt, what to refuse, and how to tell whether a tool helps the students an accommodation letter names.
What are the top 10 findings from AI in education research that instructors should know? Ten results that change ordinary teaching decisions, each with the study behind it.
How do we write and implement an institutional AI policy? Publishing a policy is the easy part, and a document alone changes nothing.

There are 37 of them in all, indexed by question rather than by topic, and there is one for students too: how should I use AI to study and learn effectively.

Search it or chat with it

Search

The box in the header searches every page as you type, and the search page does the same with room for results. Ctrl K puts the cursor in the box from anywhere on the site.

Chat

The whole knowledge base is published as machine-readable files, so you can point your own assistant at it and get answers that cite these pages instead of inventing them. Copy the prompt from the AI page, paste it as your first message, and then ask your question; a good answer names the pages it used, so you can check it. If you would rather not set anything up, two assistants already know the material: the AI in Education Advisor Gemini Gem, and a NotebookLM notebook built on the same pages. Assistants that cannot open links can be handed llms-concepts.txt, every concept and FAQ page in one file.

Read the whole thing offline

The knowledge base is also published as a book, in two formats: EPUB for a phone, a tablet, or an e-reader, and PDF for printing or for a computer. Each one holds the concept pages, the FAQs, the home introduction, the page about using the site with an AI assistant, and an appendix of free tools and resources, organized into chapters with a numbered table of contents you can click. Neither includes the article summaries, which stay on the site, and both carry the same public-domain notice and the statement of how the text was made. The download links sit on the home page and on the AI page.

Fifteen of the translations have their own edition as well, in both formats. The other seventeen link the English edition, the way their home pages do.

Read the site in your language

The site is published in 33 languages: English, plus 32 translations of the main pages, the concept pages, the FAQs, and the resources. The article summaries are English only.

Español · Français · 中文 · Deutsch · 日本語 · 한국어 · Português · العربية · עברית · हिन्दी · Русский · Українська · Italiano · বাংলা · اردو · Bahasa Indonesia · Tiếng Việt · Nederlands · Svenska · Dansk · Suomi · Norsk bokmål · Română · Magyar · Čeština · Български · Slovenčina · Hrvatski · Ελληνικά · Polski · Kiswahili · Türkçe
Each translated page carries a notice that the translation is automatic and has not been reviewed by a native speaker, and each links back to the English page it came from.
The language menu in the header switches page by page, and remembers your choice for next time.

What comes next

Issue #1 goes out tomorrow with the week's research, grouped by who has to act on it. After that, one issue a week, each with the two or three results that change what a reader should do, and a short record of what changed on the site itself.

To keep up, subscribe to the newsletter feed for the issues alone, or to the knowledge base feed for every new page as it is added. Every issue stays at the archive.