AIEd Knowledge BaseAIEd KB
On this page

Concept

Research Methods in AIED

created 2026-08-13 · updated 2026-11-01

Research methods in AIED — the set of empirical designs, data-collection strategies, and analytic techniques researchers use to study AI in education: whether and how AI tools support (or harm) learning, and under what conditions. The knowledge base's corpus spans experimental, survey, qualitative, design-based, computational-benchmark, and review methods. Each has distinct strengths and limitations, and choosing among them involves trade-offs among internal validity (confidence in causal claims), external validity (generalizability), ecological validity (real-world authenticity), and the feasibility of studying fast-moving AI tools.

Questions to Consider

  • The page's central tension: the strongest designs for causal inference (randomized experiments) are the hardest to run in real classrooms, while the most authentic settings offer weaker causal control. If you had to decide whether an AI tutor helps learning, which of these two failures would you rather live with — and why?
  • Before you read, can you name the difference between internal, external, and ecological validity? The page argues every design trades these off. How might a study that's rigorously causal still tell you almost nothing useful about a real classroom?
  • A benchmark shows an AI scores high on accuracy, but the page insists high benchmark accuracy does not entail educational effectiveness. Why might a system that 'passes the test' still fail to help students learn — and what kind of evidence is missing?
  • Design-based research iterates on a real intervention but can't attribute gains to a specific mechanism, while an RCT isolates causes but runs in artificial conditions. Given the fast pace of AI change, how long do you think a rigorous RCT remains relevant before the tool it tested is obsolete?
  • Delphi expert consensus establishes agreement among experts, not empirical effect. When is it legitimate to build a competency framework from what experts believe, versus from data about what works — and how would you tell the difference in practice?
  • The page advocates triangulation — combining benchmark evaluation, experiments, measurement, and qualitative work to judge both whether a tool works and how. Before you read, where in a claim like 'this AI improves learning' would each method be needed to make you confident?

Introduction

The central tension in AIED research is that the strongest designs for causal inference — randomized experiments — are often the hardest to run with authentic AI tools in real classrooms, while the most authentic settings (field deployments, case studies, log-data analyses) offer weaker causal control. No single method resolves this; the field advances by triangulating across methods, and by being explicit about what kind of claim each design can support. Every method also carries cross-cutting limitations — generalizability, measurement validity, the fast pace of AI change, reproducibility, and weak theory use — that readers must weigh; see Limitations in AIEd Research.

The page's subject is method rather than findings. Learning Sciences is the substantive field these methods serve: where this page covers how a study should be designed, measured and reported, that page covers what the field has established about how people learn and how learning environments should be designed, and it treats design-based and mixed methods as the learning sciences' signature approaches rather than two options among many.

Adjacent to this page is AI-Assisted Educational Research, which covers AI as an instrument of the field's own work: literature search, screening, review automation, qualitative coding, analysis, and writing. Its scope includes practitioner research such as the scholarship of teaching and learning, and it stays distinct from the designs this page describes.

Martin, Rost, Koenen and Graulich (2026) press the same scrutiny on the field's knowledge production itself, using Chang's (2004) nomic-measurement problem: measuring a quantity requires a law relating it to something observable, yet that law cannot be tested empirically without already knowing the quantity. They argue that AI-derived measurement functions, which emerge from training data and optimization rather than from the researcher, may intensify rather than resolve it. Such functions can look precise yet remain epistemically opaque, so the researcher's role shifts to interpreting and validating computational outputs. Their seven-phase reflective framework (problem framing; instrumentation and measurement; experimentation and evidence-based inference; comparisons and replication; building norms and consensus; implementation and its consequences; and continuous refinement) is offered as an analytical aid, not a validated method, and they argue comparability must extend across student populations rather than the subsets a model serves well.

Reporting rigor and the TEP-AIED model

The reporting quality of AI-in-education studies is itself a research concern. The TEP-AIED model (Hwang, Xie, Wah & Gasevic, 2026) offers a structured framework for presenting AI-in-education research with rigor, organizing the essential components of a study report — theory/technology/educational problem framing, design, data, analysis, and results — so that readers and reviewers can assess whether claims are supported and whether the work is reproducible. It responds to the field's chronic weaknesses in reporting (vague tool descriptions, unstated model versions, omitted evaluation details) that the limitations page documents. Reporting frameworks like TEP-AIED sit alongside established reporting checklists (e.g., CONSORT-style guidance for trials, PRISMA-style guidance for reviews) as part of the field's broader move toward methodological transparency and reproducibility.

RAISE (Allison, 2026) approaches the same problem from the opposite direction — as a checklist rather than a narrative structure. It sets out 30 items across ten thematic domains (educational justification and theoretical grounding, AI system specification, AI role and interaction, Accessibility and cultural fit, setting and participants, human involvement, study design and evaluation, ethics and trustworthiness, transparency and reproducibility, and limitations and implications), with an editable version and a companion Ethics and Risk Matrix covering learner agency, equity of access, data AI Governance and algorithmic transparency. The two frameworks address each other directly: TEP-AIED characterizes RAISE as comprehensive but faults its breadth, arguing that "its breadth and granularity may make it complex and less accessible for routine empirical applications," while RAISE's own framing is that it mandates no method or model and only requires that choices be made visible. Read together they mark the trade-off in this literature — the fuller audit checklist versus the leaner three-dimension narrative — and they converge on the same non-negotiables: name and version the AI system, disclose prompts and interaction design, define treatment and comparison conditions, report ethical review and risk mitigation, and state whether outcomes measure performance, retention, or transfer. For a study to be adjudicated on those terms, the reporting instrument has to be adopted at design time rather than assembled at manuscript stage, which is the point both frameworks insist on. Corpus-level historical analysis is itself a methodological choice with transparency obligations: Rismanchian & Doroudi locate each paper in their AI×Ed framework based on author judgment of abstracts and full texts, explicitly acknowledge that this is not a systematic or scalable data-driven categorization, and make their full dataset publicly available as supplementary material for replication.

Experimental and quasi-experimental designs

An efficacy study tests whether an intervention produces its intended learning effect, typically using experimental or quasi-experimental designs that compare outcomes with and without the intervention. Experiments randomly assign learners to conditions (e.g., AI tutor vs. human tutor, or AI-scaffolded vs. unassisted) to estimate causal effects on outcomes like learning gains, engagement, or motivation. Randomized controlled trials are the gold standard for internal validity. A randomized field study of human support plus AI tutoring and an RCT on generative AI in teaching use assignment to isolate causal effects. Quasi-experimental designs (pre/post, between-subjects, or matched groups without randomization) are more feasible in intact classrooms but weaker on causal claims.

Survey and structural-equation-modeling studies

Cross-sectional surveys measure self-reported attitudes, perceptions, motivation, Self-Efficacy, and technology acceptance, often modeled with regression or structural equation modeling (SEM/PLS-SEM) to test hypothesized relationships and mediators. These dominate the knowledge base's corpus, particularly for acceptance, motivation, and psychological-mechanism questions.

  • Strengths: large samples; broad, low-cost coverage; can test complex mediational models of psychological mechanisms; feasible for studying attitudes that are hard to observe.
  • Limitations: cross-sectional data cannot establish causation; common-method/self-report bias; convenience sampling limits generalizability; mediators inferred from covariance, not manipulation.

The instrument itself deserves separate scrutiny. What a questionnaire, interview, or diary can and cannot establish — and the documented gap between what people report and what they do — is gathered on Self-Report Measures.

Qualitative methods

Interviews, focus groups, and thematic analysis produce rich, contextual accounts of how students and teachers experience AI tools, the meanings they attach to them, and the tensions and harms that standardized measures miss. Research on AI tutor safety and how AI changes teaching workflows rely heavily on qualitative evidence. See the dedicated Qualitative Research concept page for the full treatment of qualitative approaches — thematic analysis, grounded theory, phenomenology/phenomenography, discourse analysis, observations and ethnography, case studies, and interviews/focus groups — each with knowledge base exemplars.

Mixed-methods designs

Mixed-methods studies combine quantitative and qualitative strands — often sequentially (e.g., QUAL→QUAN→qual) — so that qualitative data explains or contextualizes quantitative findings. A mixed-method study of GenAI and sustainable learning pairs three-wave surveys with educator interviews; the competence-paradox study uses instructor focus groups, a student survey, and follow-up interviews.

Design-based research (DBR)

DBR iteratively designs, implements, and refines an educational intervention in authentic contexts, cycling between theory, design, and real-world practice. It is prominent in the knowledge base for developing AI learning environments and pedagogical models. See the dedicated Design-Based Research concept page for the full DBR cycle, exemplars, and its strengths/limitations. A canonical AIEd example is the AI-Assisted Collaborative Learning model study (Putra et al.), which ran a four-phase DBR cycle — needs analysis, model design, eight-week classroom implementation, and model refinement — iterating on a four-stage learning cycle (problem identification → AI-assisted collaborative inquiry → collaborative Problem Solving → reflection and presentation). Other exemplars develop AI Literacy teacher training (Development and evaluation of artificial intelligence literacy training for teacher education students) and GenAI Scaffolding for critical thinking (Scaffolding critical thinking with generative AI: Design principles for integrating large language models in higher education).

DBR trades the causal control of experiments for ecological authenticity and iterative refinement: it is the right tool for "how do we design this AI learning environment to work in practice?" questions, and its evidence is strongest as proof-of-concept and design guidance rather than causal efficacy. Reading DBR learning gains requires the same caution as other designs — without an unassisted, controlled outcome measure, gains can reflect the same AI-inflated-performance confound documented under learning gains.

Systematic reviews and meta-analyses

Reviews synthesize the evidence base rather than running a new experiment. Systematic and scoping reviews apply a transparent protocol to search, screen, appraise, and synthesize a body of studies; meta-analyses additionally pool effect sizes across studies to produce a weighted summary estimate and test moderators. A comprehensive ITS review and a systematic review of GenAI in higher education exemplify the approach.

See the dedicated Meta-Analysis and Systematic Review concept page for a fuller treatment of systematic review and meta-analysis in AI in education — including their relationship to primary designs, PRISMA reporting, and their strengths and limitations.

Computational and benchmark evaluation

Computational evaluation assesses AI systems directly — against benchmarks, ground-truth labels, or human judgments — rather than studying human learners. This includes benchmarks, Large Language Models (LLMs)-as-judge approaches. This is the closest method to AI Ed Evaluation (see the distinction below).

Other designs: longitudinal, case, and simulation studies

Beyond the major families, the knowledge base uses longitudinal designs that track learners over time (a longitudinal LMS study), case and in-the-wild studies of authentic usage (large-scale analysis of real student interactions), and simulation studies in which LLMs stand in for students or patients (LLMs as simulated learners, Simulation). These trade breadth or control for realism and for access to phenomena that are otherwise hard to observe.

Expert-consensus methods: the Delphi technique

The Delphi method is a structured technique for establishing expert consensus on a question where the answer is not yet known empirically — most often used in the knowledge base to develop frameworks, competency lists, and definitions that practitioners and researchers can agree on. In a Delphi study, a panel of experts responds to successive rounds of questionnaires; after each round, an anonymized summary of the group's responses is fed back, and experts revise their answers until the group converges on agreement (typically defined by a pre-set threshold, e.g., 75%). It is a way to build construct validity and professional consensus through iterative, anonymized consultation rather than a single survey or vote.

  • Strengths: produces consensus from a diverse expert panel without in-person group pressures (anonymity reduces dominance effects); well-suited to defining constructs, competencies, and frameworks when no validated measure exists; iterative rounds let experts refine and converge; feasible where full experiments or large samples are impractical.
  • Limitations: consensus reflects expert judgment, not empirical evidence — it establishes agreement, not effect; results depend on panel composition and the (subjective) consensus threshold; can be slow across multiple rounds; a single panel's judgment may not generalize.
  • Exemplars: the SAIL framework study (three rounds, 17 experts, refining AI-literacy competency levels), the HCAP framework study (three rounds, 30 teachers, defining 25 AI-teacher competencies), the AI Literacy Heptagon (which used expert input/consensus alongside a PRISMA-guided review), and.

Delphi is often combined with other methods — for example, expert consensus can be used to validate a framework (as in SAIL and HCAP) that is then tested or implemented via design-based research or survey studies. It sits alongside qualitative and expert-judgment approaches and contributes to the validity of framework-based instruments.

Research vs. evaluation: connections and distinctions

Research and evaluation are closely related but distinct. Research asks generalizable questions about how AI affects learning — "does scaffolding improve learning outcomes?" — and aims to build theory and evidence that transfers beyond the specific study. Evaluation (see AI Ed Evaluation) assesses whether a specific AI tool or system works — is accurate, reliable, pedagogically sound, and fit for purpose — against benchmarks, rubrics, or stakeholder-defined criteria. Research emphasizes internal validity and generalization; evaluation emphasizes system quality and local decision-making.

The boundaries blur: benchmark studies are evaluation that can feed research, and evaluation instruments (rubrics, ground-truth sets, validity frameworks) depend on the Educational Measurement and Assessment Validity concerns that research clarifies. Conversely, research findings on what supports learning should inform how AI tools are evaluated. The knowledge base treats them as complementary: computational and benchmark evaluation (Benchmark, AI Ed Evaluation) tells us whether an AI system is technically sound, while efficacy and survey research (RCT) tells us whether it helps people learn.

Choosing among methods

Method choice follows the research question. Causal-effect questions favor experiments (RCT); mechanism and perception questions favor surveys and qualitative work; system-quality questions favor computational evaluation (Benchmark, AI Ed Evaluation); synthesis questions favor reviews and meta-analyses; design questions favor DBR; and questions about what experts agree a construct, competency, or framework should contain favor expert-consensus methods like the Delphi technique. Given the field's heterogeneity and the speed of AI change, the knowledge base's corpus reflects a deliberate move toward triangulation — combining computational evaluation with efficacy, qualitative, and expert-consensus evidence to judge both whether a tool works and whether it helps learning.

Equally important is reading any single study with awareness of the cross-cutting limitations that affect AIED research as a whole — methodological constraints, the fast pace of AI change versus slow publication, reproducibility and FAIR-practice gaps, reliance on proprietary tools, and weak or uncritical theory use. See Limitations in AIEd Research.

Contrasting the major research traditions

The three major research traditions — quantitative, qualitative, and experimental — differ fundamentally in what they can claim, what they sacrifice, and when each is appropriate. Understanding these contrasts is essential for both designing and reading AI-in-education research.

What each tradition establishes

Dimension Quantitative / survey Qualitative Experimental
Core question How much? How related? What does it mean? How is it experienced? Does X cause Y?
Primary data Numbers, scales, self-report Words, observations, artifacts Outcome measures across assigned conditions
Inference target Patterns, correlations, mediation Meaning, mechanisms, categories Causal effects
Internal validity Weak (correlational) Weak (no control) Strong (random assignment)
External validity Strong (large samples) Limited (small, context-bound) Moderate (controlled conditions)
Ecological validity Moderate High Lower (artificial conditions)
  • Quantitative research measures and models relationships among variables — surveys, SEM/PLS-SEM, measurement, longitudinal tracking. It provides breadth, precision, and generalizability but cannot establish causation from cross-sectional data and inherits measurement limitations (including self-report bias).
  • Qualitative research interprets meaning and experience — interviews, focus groups, thematic analysis, grounded theory, phenomenography, discourse analysis, observation/ethnography, case studies. It provides depth, mechanism, and theory-building (see Theory Development in AI in Education) but limited generalizability and weak causal support.
  • Experimental and quasi-experimental designs (see RCT) estimate causal effects via random assignment or matched comparison — the gold standard for internal validity, at the cost of cost, speed, and ecological validity.

Quantitative work depends on Educational Measurement — reliable, valid instruments for the constructs being studied. Qualitative work reveals the mechanisms and meanings those instruments may miss. Experimental work estimates whether an intervention causes the outcomes the instruments measure. The three are complementary layers: instruments quantify constructs, experiments establish causality, and qualitative work explains the how and why behind the numbers.

Mixed-methods designs intentionally combine quantitative and qualitative strands so their strengths offset each other's weaknesses — quantitative breadth plus qualitative depth, with triangulation increasing confidence.

Usability and HCI research

A distinct methodological strand — usability and HCI research — evaluates how users interact with an AI system: its usability, usefulness, learnability, and user experience, using think-aloud protocols, structured user studies, interviews, and observation. It is the closest to AI Ed Evaluation and answers a prerequisite question: even a pedagogically sound tool fails if it is unusable. Usability research shares data-collection methods with qualitative research but aims at evaluating an artifact rather than interpreting meaning.

Benefits and limitations across traditions

  • Quantitative/survey: benefits — large samples, broad coverage, tests complex mediators, efficient. Limitations — no causation, self-report bias, convenience sampling, instruments may measure the wrong construct.
  • Qualitative: benefits — deep insight, surfaces unexpected phenomena and harms, essential for theory-building, centers under-represented voices. Limitations — limited generalizability, researcher dependence, small samples, weak causal support, hard to synthesize.
  • Experimental: benefits — strongest causal inference, clean outcome measurement, effect-size estimation. Limitations — costly/slow, artificial conditions, fast-changing AI dates results, underpowered small samples, ethical constraints.
  • Mixed-methods: benefits — triangulation, breadth + depth, explains unexpected results. Limitations — complex, resource-intensive, integration can be shallow, inherits each strand's weaknesses.
  • Usability/HCI: benefits — identifies adoption barriers, actionable design guidance, fast and cheap. Limitations — does not establish learning effects, small samples, self-report satisfaction can mislead.

In practice, AI-in-education research rarely falls cleanly into one tradition. The strongest evidence triangulates: a computational or usability evaluation establishes that a system works, an experiment establishes that it causes learning, quantitative instruments measure the constructs, and qualitative work reveals the mechanisms and meanings — together answering both whether a tool helps learning and how and why.

Bulut (2026) adds a bibliometric strand that sits apart from these four: mapping 407 records by co-word and co-authorship network to trace a field's conceptual development. The author is explicit that the SWOT reading built on top of it is an interpretive device rather than an independent analytical technique, which marks the boundary between synthesis and analysis in this tradition.

  • "A statistic can be trusted once it is computed." The benchmark checks the check first. Every probe and every verdict except six named sets of statistics was run on simulated graders with a planted effect and without one, and a measure whose check did not find the planted effect was reported as unmeasured rather than as a result. Its reports are generated from their own markdown and code, so every printed number links to the file and field it came from and is recomputed at build. A Verdict is a design decision that can be tested before any real result is read.

Common Misconceptions

"High benchmark accuracy shows educational effectiveness." It does not, and the gap is the field's central measurement problem: a system can score high on accuracy while producing no learning, because benchmark accuracy measures task performance under the conditions the benchmark set. Educational effectiveness requires a learner outcome under conditions the technology does not control.

"One research tradition is methodologically superior." The traditions trade one limitation for another by construction. Design-based research iterates on a real intervention but cannot attribute gains to a specific mechanism, while an RCT isolates causes but runs in artificial conditions. Choosing between them is choosing which claim you can make, not which method is better.

"A lenient measurement rule is a minor methodological choice." It changes the finding. Crediting any completion predicted later unaided performance barely above a skill-difficulty constant (pooled AUC 0.604 against 0.595) while declaring 93.9% of student-skill pairs mastered against 72.8% under a strict rule. The rule produced a 21-point difference in who was declared ready and almost no predictive improvement to show for it.

"Statistical significance settles whether an intervention worked." It settles whether an effect was detectable. Whether it was a learning effect depends on the outcome measure, the comparison and the conditions under which it was taken — which is why the design and reporting choices in this page's sections carry as much weight as the analysis.

Connected Concepts

Connected Articles

Connected FAQs

Metadata

Study designExperiment
LevelHigher Ed
Assessment and measurementEducational Measurement
Research methods and evaluationAI Ed Evaluation, Benchmark, RCT, Research Methods in AIED
Foundations of AI in educationAI in Education
Page kindEvaluation
SourceHistoryEditChat

Embed this page

Copy the code below to embed this page in a learning management system or other website. The embedded version shows only the page content — it hides the site header, navigation, and footer.