Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth

Synthesis: This study challenges the standard practice of evaluating LLM qualitative coding by agreement with human coders, using data from a K-12 AI platform: five LLM systems and three trained human coders applied a 72-item hierarchical codebook to 2,560 educator messages.

An independent domain expert judged 855 pairwise code-set comparisons blind to source, treating human and machine outputs symmetrically. Human-LLM agreement (mean Jaccard 0.30) fell well below human-human agreement (0.52), yet the blind verifier preferred human and LLM coding at indistinguishable rates (51.5% vs 48.5%, p = 0.537).

A Bradley-Terry ranking placed two LLMs above two of three human coders, and for several substantive codes human consensus encoded shared bias that the verifier rejected in favor of the LLM interpretation — evidence that agreement metrics can mislead automation decisions.

The study contributes a transferable blind-verification protocol for evaluating qualitative coding quality in AIED research, with implications for how LLM-assisted analysis of educator and student data should be validated.

What this means for practice

• Researchers. Stop treating high agreement with trained human coders as sufficient evidence that LLM coding is correct; adopt a blind verification protocol in which human and machine outputs are judged symmetrically by an independent domain expert.
• Researchers. Delegate by code rather than wholesale: this study classified 6 of the 72 codebook items as automatable, 12 as better served by LLM coding, 15 as demonstrably requiring human expertise, and 16 as suited to confidence-based triage with a calibrated model.
• Instructors. When LLMs are used to analyze educator or student messages at scale, reserve human review for categories that demand contextual inference and treat human consensus as fallible — three trained coders agreed on some codes while both under-applying them.
• Designers. Build code-level routing into analysis tooling, so that each code runs under the oversight level the evaluation assigns it instead of a single uniform human-review setting.

Limitations

• The study employs a single 72-item codebook on a single dataset of 2,560 K-12 educator messages, so the specific codes identified as automatable or human-required may not generalize to other domains, data structures, or coding approaches.
• Verification rests on one independent expert, who judged 855 pairwise comparisons and reached a decisive preference in 801 cases (93.7%); with no second verifier there is no estimate of inter-verifier reliability, and a different expert might have endorsed the human coders' interpretation.
• The LLM rankings are a capability snapshot: all inference ran at temperature zero with no session memory, so enhanced prompting strategies were left unexplored and specific ordering is expected to shift across model generations.
• Only five of the models that coded the corpus entered verification, selected under a fixed verification budget, so the aggregate finding of no overall human–LLM preference is conditional on that model mix.

Citation

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement is not quality: Blind expert verification of human and LLM qualitative coding.