Authors: Kamila Misiejuk, Sonsoles López-Pernas, Eduardo Araujo Oliveira, Brendan Eagan, Mohammed Saqr Source: Computers and Education: AI, Vol 11 — Open Access (CC BY 4.0)
Key Findings
Critical methodological paper on using LLMs for automated qualitative coding of ordered data (where code order matters). Presents two evaluation approaches for ordered coding quality. Demonstrates systematic and statistically significant differences between LLM and human coding across structural, transitional, and code-level metrics. Warns that LLM coding errors can propagate through automated feedback systems, amplifying inaccuracies. Uses consistent context window prompting method.