Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education

Synthesis: The AIED community over-relies on inter-rater reliability (IRR) — typically a single Cohen's κ coefficient — as a mechanical gatekeeper for "ground truth." This practice is insufficient and potentially misleading for the complex, noisy realities of educational data. The authors propose four practical shifts to strengthen the validity and reliability of the evidence base of labeled AIED datasets.

Core Argument

The Problem

Noise vs. Bias in Educational Labeling

Human judgment is subject to both noise (random variability) and bias (systematic directional error). While bias and fairness have received extensive attention, noise is an underexamined obstacle in AIED. Education is inherently noisy — assigning grades, defining engagement, and identifying giftedness all involve subjective interpretation.

Why κ Alone Fails

Educational settings present specific challenges that undermine threshold-based IRR heuristics:

• High-inference constructs: Complex concepts like "engagement" or "understanding" require deep judgment beyond surface-level agreement
• Skewed label distributions: Class imbalance inflates or deflates κ in misleading ways
• Temporally segmented Multimodal AI data: Video, audio, and log data over time create segmentation challenges where raters may agree on the label but disagree on when to apply it

The LLM Annotation Risk

The growing use of LLMs as annotators introduces new threats:

• Automation bias: Over-trusting Large Language Models (LLMs) output without verification
• Circular validation: Using the same model to evaluate its own output
• Silent degradation: LLM annotators may produce deceptively consistent but subtly wrong labels

The Four Shifts

1. IRR as Diagnostic, Not Gatekeeper

Stop treating κ > 0.8 as a binary stamp of approval.

Instead, use IRR to localize disagreement — identify where and why raters disagree, then refine constructs and codebooks accordingly. Disagreement is information, not failure.

2. Transparent Annotation Reporting

Require thorough documentation of:

• Rater expertise and training: Who labeled the data and what qualifies them?
• Codebook development: How were categories defined and iterated?
• Reconciliation procedures: How were disagreements resolved — majority vote, discussion, expert arbitration?
• Segmentation rules: For multimodal/continuous data, how was it divided into labelable units?

3. Mitigate LLM Annotation Risks

• Bias audits on LLM annotators before deployment
• Verification workflows with human-in-the-loop checks
• Guard against automation bias and circular validation through independent evaluation

4. Complement Agreement with Validity Evidence

Go beyond agreement statistics with:

• Uncertainty-aware labeling: Assign multiple labels to the same item to capture ambiguity and nuance
• Criterion-related checks: Predictive tests to verify labels forecast intended educational outcomes
• Close-the-loop evaluations: Test whether systems trained on these labels actually improve learning beyond a reasonable control condition

Case Studies

The paper illustrates these shifts through case studies of multimodal tutoring data, demonstrating how the four-shift framework applies to real AIED annotation challenges.

Practical Recommendations

1. Always report multiple IRR metrics (κ, α, percentage agreement) and discuss their limitations given the data characteristics
2. Make codebooks and annotation guidelines public whenever possible
3. Treat LLM annotations as hypotheses to verify, not as ground truth
4. Include at least one validity check beyond agreement in every labeled dataset paper
5. Design annotation workflows that surface ambiguity rather than forcing binary decisions

Open Questions

• What is the "minimum viable validity evidence" for different AIED use cases (high-stakes assessment vs. formative practice)?
• How should the four shifts be adapted for fully automated annotation pipelines with no human raters?
• Can uncertainty-aware labeling be standardized across the field, or is it inherently task-specific?

What this means for practice

• Researchers. Report the whole annotation workflow — rater expertise and training, codebook development, how disagreements were reconciled, and how continuous multimodal data was segmented — rather than a standalone coefficient, since the authors note many studies fail to report internal reliability at all.
• Stop treating κ > 0.8 as a gate. Read IRR as a diagnostic that localizes where raters disagree, and use that disagreement to refine the construct rather than discussing cases until the coefficient reaches 0.80.
• Audit any LLM annotator before trusting it and keep an independent verification step in the loop: the paper cites evidence that showing annotators LLM pre-labels reduced their accuracy when the model was wrong.
• Add at least one validity check beyond agreement to every labeled-dataset paper — multilabel annotation with confidence levels for ambiguous items, a predictive test that the label forecasts a distinct outcome, or a close-the-loop test of whether a model trained on the labels improves learning over a control condition.
• Budget domain expertise into annotation design. For high-inference constructs such as engagement, minimally trained crowd-workers are unlikely to approximate ground truth; the paper points to certified-observer protocols such as BROMP as the standard to normalize.

Limitations

• This is a position and theoretical analysis paper with no new dataset and no experiment, so it argues the four shifts should improve reliability and validity rather than showing that they do.
• Its concrete illustrations come from multimodal tutoring case examples, such as a transcript coded for student math errors through explicit visual and implicit verbal lenses; the prevalence of κ-threshold misuse is evidenced by cited cases rather than a systematic sample of AIED papers.
• The paper sets no quantitative bar for sufficiency: it leaves the "minimum viable validity evidence" for different AIED use cases as an open question and does not resolve how the four shifts apply to fully automated pipelines with no human raters.
• The automation-bias and LLM-annotation risks it raises are drawn from other researchers' studies, so those claims are not tested in this work.

Citation

Thomas, D. R., Borchers, C., Vanacore, K. P., Koedinger, K. R., & Kizilcec, R. F. (2026). Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education. Accepted to AIED 2026.