On this page

Your co-author wants a paragraph in the methods section about the AI tools the team used. Or the screening pile is longer than the deadline allows. Or a reviewer has asked, politely, whether a person read the transcripts, and you are not sure what the honest answer is.

The short answer for that meeting: AI can take search, a first screening pass, coding triage, and the drafting, and it cannot carry responsibility for what the project concludes. Decide which tasks are open to it before the project starts, verify outputs against a human standard with a named person adjudicating, and log enough that an outsider could reconstruct what happened. Most teams find the verification and logging hours are work added rather than removed.

The short version

1. Write down which research tasks may use AI and which may not, before the first search runs. Sort by stakes rather than by tool.

2. Put the automation on screening. Keep data extraction, reconciliation, and synthesis with people.

3. Settle the codebook and the rubric before an automated pass touches the data.

4. Decide what verification means and who signs it. Agreement between a model and a coder is a similarity measure, not a quality result.

5. Verify every reference by hand, authors first, whenever AI touched the writing.

6. Log prompts, model versions, and corpus versions, and budget the hours for it.

Decide what AI may touch before the project starts

The scope decision belongs in the project file rather than in someone's memory. In Dai and Chan's (2026) focus-group study, researchers who had thought about it calibrated use by stakes and intellectual centrality: heavier on low-stakes procedural work, cautious where the scholarly contribution was central. Twenty-seven of the 28 postgraduate researchers in that study had used GenAI somewhere in the workflow, including academic writing and translation, so for most teams the question is where the line sits rather than whether one exists.

Write the line as a list with a verdict for each task: literature search, abstract screening, extraction, qualitative coding, statistical analysis, drafting, reference formatting. Read that study as a map of what researchers reported doing. It is a focus-group account, not an effectiveness result.

Put the automation on screening and keep synthesis in the team

Systematic reviews are where automation has advanced furthest. Tools such as ASReview, SWIFT-Review, Covidence, AIScreenR, and MetaMate cut the procedural burden mainly at abstract screening, while data extraction, reconciliation, and synthesis stayed with human reviewers in the one team that reported its workflow (Wang et al., 2026).

Most of the benefit comes from two design decisions. Write decision rules for gray-area cases before screening starts, and keep a "maybe" category with a shared adjudication log so the same judgment gets applied to the same kind of record. Without the log, the second reviewer's rule and the first reviewer's rule diverge quietly, and the exclusion list stops being reproducible.

Fix the codebook before the model reads anything

Qualitative analysis is the stage where interpretation is the product. Constructs, coding rubrics, and training data are human decisions rather than model outputs (Martin et al., 2026), which means the codebook has to be settled before an automated pass runs over the corpus. The alternative is a coding scheme written around what the model happened to surface, and the themes then describe the tool's reading of your data.

Delegation can be finer-grained than all-or-nothing. One blind-verification study classified 15 of 72 codebook items as demonstrably requiring human expertise, 12 as better served by a model, and 16 as suited to confidence-based triage. Sort your own items that way and the labor saving lands where it is safe.

Topic modeling can extend that structure without taking over interpretation. Ober, Courey and Flor (2026) combined grounded coding, SentenceBERT embeddings clustered with Affinity Propagation, and GPT-4 labeling on six focus groups with 13 teachers. Human review dropped 18 communication clusters and 16 digital-literacy clusters that were conversational filler, leaving 14 and 16 meaningful clusters; ChatGPT-4o and Copilot agreed on 33.3% and 43.8% of labels, with cosine similarity 0.835–0.880, and human review found no clear hallucinations. The study is a worked pipeline, not a comparison: it has no human-only or AI-only arm, and 13 teachers cannot establish generalizability. Use it to see the division of labor, not as evidence that the hybrid beats alternatives.

Decide what verification means and who signs it

High agreement between a model and a human coder gets reported as if it established correctness. It does not. Liu et al. (2026) had an independent expert judge 855 pairwise code sets blind to source. Human-Large Language Models (LLMs) agreement (mean Jaccard 0.30) sat well below human-human agreement (0.52), yet the blind verifier preferred human and machine coding at rates it could not distinguish (51.5% against 48.5%, p = 0.537). In some cases human consensus encoded a shared bias the verifier rejected in favor of the model.

So the verification step needs a gold standard that is neither the model's output nor the coder's agreement with it, plus a named person who adjudicated the disagreements. Report both. A similarity statistic describes how closely two readings match, and it says nothing about whether either reading is right.

Search fast, then check every source yourself

Generative search summarizes, recommends, synthesizes, and converses, which unsettles the older arrangement in which the system locates and the reader interprets (Report on CHIIR 2026 Workshop on Generative AI and Academic Search (GAI&AS)). Treat it as a discovery tool. The workshop report carries two cautions: capability is uneven, with highly cited scholars reconstructed at roughly twice the rate of lower-cited peers, and a librarian described a trust gap in which students over-trusted generative search while faculty distrusted it.

Reference verification is where this becomes non-negotiable, and the failures cluster in the author field, the one that carries credit. Denny et al. (2026) audited 723,930 publications and 15,872,533 references and verified 30 fabricated references across 14 computing-education papers, all from 2025 and 2026. Seventeen of the 30 were hybrids: a real title attached to fabricated or incorrect authors. The verified count at one technical symposium rose from 3 in 2025 to 17 in 2026, in 2.3% of that year's proceedings papers. Their total is a deliberate lower bound, and automated checkers inherit the flaws of the metadata they treat as ground truth. Check authors first.

Treat an automated score as a hypothesis to test

Automated measurement functions can look precise and predictive while remaining opaque. Martin et al. frame this with Chang's nomic-measurement problem: measuring a quantity requires a law relating it to something observable, and that law cannot be tested without already knowing the quantity. AI-derived measurement functions emerge from training data and optimization rather than from the researcher, so explainability work matters for showing whether a model tracks semantic understanding or only keywords.

The failure is concrete. In an evaluation of eight AI teaching agents, the agent ranked third by the platform's own score ranked last on an expert-validated rubric. Human-machine disagreement is systematic rather than random, and rubric operationalization often matters more than prompt craftsmanship (Tseng et al., 2026). Before an automated score becomes your instrument, validate it against a measure tied to the capability you intend to study. For the measures themselves, see What Measures and Research Methods Can an Instructor Use to Evaluate AI-Related Interventions?.

Write the method section so an outsider can audit it

Adoption has run ahead of reporting. Zabaleta and Lin (2026) analyzed 888 review-automation papers: since 2023, 38.0% of software and product papers reported no evaluation at all, against 9.3% of LLM papers, and model access was overwhelmingly proprietary (84.1%). A favorable overall assessment did not imply fitness for delegation either, since 52% of 118 positive-only LLM papers still reported a concern that the workflow fell below the bar required for its role. Their PRISMA-LLM framework separates implementation disclosure from consequence-sensitive evaluation and treats its five levels as disclosure tiers rather than risk tiers.

The practical version: name the system and its version, the prompts, the corpus version, what a human checked, and where it failed. Keep prompt-and-response logs, because a fluent generated theme list looks inevitable long before its evidentiary work has begun (Holster, 2026).

Budget the hours as well. A 115-day, single-investigator case study of a persistent research agent found the strongest pattern was capacity expansion rather than proven labor substitution: as memory and procedures accumulated, the scope of delegated work grew while the human's input did not shrink.

Run practitioner inquiry with the same rules at smaller scale

If you are a lecturer studying your own course, you are working in the corpus's thinnest strand. AI-Assisted Educational Research states this as a gap rather than a finding: AI-assisted practitioner inquiry is plausibly widespread and almost undocumented, and the review-automation and bibliometric evidence does not settle how a lecturer should study their own teaching.

What carries across is a set of dispositions:

Use AI where your local setting is not the variable. Searching your field's literature, transcribing and summarizing your own recordings, and drafting instruments or consent language all qualify.

Keep the interpretive moves. What counts as a theme, what a student's comment means, and what your classroom evidence warrants are yours to make, and the write-up should say which moves the model made and which you made (Holster, 2026).

Predefine your criteria and keep them visible. A small local study cannot recover from a construct defined after the data arrived.

Report the AI's role, the verification you did, and the limits of a single-course, single-investigator design. A workflow account is not an efficacy result.

Treat simulating a cohort as an advanced option rather than a default. Simulated learners tend to cover the easiest quadrant of real student behavior and are rarely validated after use (Simulating Students). Read How Do We Make a Simulated Student Behave Like a Real Learner? before you trust a simulator's verdict.

What researchers get wrong about this

Agreement with a human coder means the coding is right. Human-LLM agreement in the blind study was 0.30 Jaccard against 0.52 for human-human, and the blind verifier could not tell the two coding sources apart at 51.5% against 48.5%. Agreement records that two readers converged, including when they converged on the same bias.

A reference checker replaces checking references. The 30 fabricated references in the Denny audit include 17 hybrids where the title was real and the authors were not, which is exactly the case a checker trained on metadata handles worst, since the metadata is the thing that is wrong.

A favorable evaluation makes a workflow fit for your role. Of 118 positive-only LLM papers, 52% still reported a concern that the workflow fell below the bar its role required. Read the depth of the evaluation against the job you are giving it.

Prompt craft is the main lever on quality. Rubric operationalization often matters more than prompt craftsmanship in human-machine agreement, and the eight-agent evaluation ranked the same system third and last depending on whose rubric was applied.

AI takes work out of the project. The 115-day case study found delegated scope growing as the agent's memory accumulated. Verification, adjudication, and logging are the hours that grow with it.

What will come back at you

"My co-author already does this and nobody has complained." Nobody complaining is not a finding about validity. The disclosure question is separate from the quality question, and the 38.0% figure for unevaluated software papers says the field's silence is widespread.

"Two models agreed with each other, so the coding is settled." Two models sharing training data share their errors. The Liu study's blind verifier preferred machine coding at 48.5%, and what made that number interpretable was an independent judge, not the agreement itself.

"We cannot audit a proprietary model, so there is nothing to report." PRISMA-LLM asks for implementation disclosure as a tier separate from evaluation depth. System, version, prompts, what humans checked, and where it failed are all reportable for a closed model.

"The ethics committee will never approve this." Model access was proprietary in 84.1% of the papers examined, which means student text typically leaves your institution. Collect only what the educational purpose needs, check whether learner data trains the vendor's models, and prefer local or synthetic data where sensitivity is high. How Should AI in Education Research Incorporate Equity, Accessibility, Privacy, Ethics, and Pedagogical Safety? covers the obligations in full.

"This will be slower than doing it by hand." For a small project it often is. The case for it rests on scope at scale, and the case against overstating it rests on the same 115-day account.

What to do this week

1. Write the task list with a verdict beside each item, and put it where the whole team can see it.

2. Add the gray-area decision rules and the adjudication log to your screening protocol before the next batch starts.

3. Pick one automated output you already rely on and validate it against a human judgment on a sample you have not looked at yet.

4. Read the last manuscript you submitted and check every author name in the reference list by hand.

5. Add a "where the AI failed" line to your methods template, so the disclosure habit is written down once.

Where to go next

The companion questions are What Measures and Research Methods Can an Instructor Use to Evaluate AI-Related Interventions? on how to measure an intervention, What Are Best Practices for Reporting and Interpreting AI in Education Research? on reporting the system and your own AI use, and What Are Notable Gaps in the Research Literature on AI in Education? on where the literature is thin. How Should AI in Education Research Incorporate Equity, Accessibility, Privacy, Ethics, and Pedagogical Safety? covers student data, and AI-Assisted Educational Research is the full concept page behind this one.

Connected FAQs

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.