A Taylorizable Process for Textual Detector Development for Research-Practice Partnerships

Synthesis: Baker, Mills, Lan, Lehman, and Barany (2026) argue that textual detectors of cognitive constructs, the automated measures of essays, short answers, discussion posts, and dialogue that increasingly sit inside deployed learning platforms, must be built with more rigor and less cost than current practice allows. Their core claim is that LLM-as-a-judge development, which replaces supervised machine learning with rubric-based prompting, is fast but often skips practices that educational data mining and automated essay scoring have long required, while fully rigorous development stays slow and expensive. Drawing on Frederick Taylor's principles of scientific management, the authors decompose detector construction into a standardized seven-step workflow: dataset selection, construct definition, codebook drafting, human inter-rater checking, category refinement, detector building and evaluation, and application at scale. The process is designed to deliver four forms of rigor, namely assessment of goodness, generalizability, construct validity, and auditability. Framed as a learning engineering contribution to research-practice partnerships, it aims to let mixed-expertise teams produce trustworthy detectors fast enough to keep pace with co-design cycles.

Key Findings

• Textual detectors of cognitive constructs are shifting from research endpoints to embedded components of deployed platforms, which makes inaccurate measurement a potential source of harm to real learners.
• LLM-as-a-judge development replaces supervised learning with rubric-based prompting, cutting effort, but is often deployed without validation on human-coded labels, leaving construct validity and reliability in doubt.
• The authors propose a seven-step Taylorizable process: dataset selection, construct decisions, codebook drafting, human inter-rater checking, category refinement, LLM detector development and evaluation, and application at scale.
• The process targets four forms of rigor: assessment of goodness, generalizability, construct validity, and auditability, with student-level cross-validation and subgroup bias checks as minimum standards.
• Taylorization standardizes each step so less specialized staff can execute it, concentrating expert judgment mainly in construct definition and definition review, which the authors argue makes that judgment visible rather than invisible.
• Cost motivates the work: a 2015 analysis found affect detectors cost about $13,490 each at ASSISTments, over$80,000 in total, and $12,460 each at Inq-ITS.
• The authors also offer six criteria for judging any detector development process: efficiency, reduced upfront planning, fewer derailments, rigor, attention to ethics, and reuse across domains.

The Rigor Gap in LLM-Based Detection

Textual detection matters because many cognitive constructs, including confusion, argumentation quality, critical thinking, and dialogue moves, are reflected in the text learners produce. The traditional route, supervised machine learning over human-labeled data, is costly: one ingredients-method analysis found affect detectors costing about $13,490 per detector at ASSISTments, more than$80,000 in total, and $12,460 per detector at Inq-ITS. LLM-as-a-judge promises relief by prompting a model against a written rubric rather than training a classifier, but the authors argue teams adopting it often skip validation on human-coded labels. The risk is concrete: an evaluation of Wisconsin's dropout early warning system, covering roughly 215,000 students and a decade of outcomes, found reasonable overall accuracy with much worse calibration for some groups, de-prioritizing students from historically disadvantaged groups.

A Seven-Step Taylorizable Workflow

The proposed process decomposes detector development into standardized steps with named owners. Step 1 selects the dataset, with the domain expert judging contextual relevance, a qualitative coder judging codeability, and practitioners judging feasibility. Step 2 decides the constructs, combining expert theory review with bottom-up and theory-driven codebook discovery run through an LLM. Step 3 drafts codebook definitions with explicit inclusion and exclusion rules. Step 4 runs human inter-rater checking on a shared subset, using metrics such as Cohen's or Fleiss's kappa, Krippendorff's alpha, or Shaffer's rho. Step 5 refines categories when agreement falls below the cutoff set in advance, and steps 4 and 5 alternate until reliability is reached or the construct is redesigned or dropped. Step 6 builds and evaluates detectors against the final human codes, and step 7 applies validated detectors at scale inside the platform.

Four Forms of Rigor

Each step is meant to produce evidence for one of four forms of rigor. Assessment of goodness asks whether the detector separates the construct from its absence, evaluated on human-labeled cases with chance-corrected metrics rather than exact-match agreement. Generalizability asks whether it holds across the full span of intended cases, which at minimum means student-level cross-validation, or prompt and retrieval development on one set of students and testing on another, plus content held out and subgroup error analysis. Construct validity asks whether the detector measures the intended construct, checked through expert review of definitions, convergence with established measures, and close reading of successes and failures. Auditability asks whether decisions can be inspected by outside reviewers, which requires documenting rubrics, prompts, model versions, and coding records, and prompting models to explain their codes while treating those explanations as hypotheses rather than faithful accounts of internal reasoning.

What this means for practice

• Research-practice partnership teams. Use the seven-step sequence as the plan for the next detector, attaching target dates and the named personnel each step requires, so research stays inside the co-design cycle rather than arriving after designs are piloted.
• Teams auditing an existing pipeline. Apply the six criteria as a rubric to locate where a process breaks down, and treat the catalogue of per-step pitfalls as a pre-mortem checklist reviewed on entering and leaving each step.
• Team leads. Keep domain expertise at construct definition and definition review, and document why decisions were made, since that record preserves expert reasoning that later readers would otherwise reconstruct.
• Measurement and ethics. Check subgroup calibration and error rates before deployment, and document false positive and false negative rates so educators can judge when automated codes should be accepted, questioned, or overridden.
• Scoping. Where calendar time is the binding constraint rather than cost, adopt the existing-data variant, and plan for local open models, since commercial models can be deprecated and local execution reduces Privacy risk.

Limitations

• This is a framework paper, not an evaluation: the process has not yet been tested in published work, and the authors report that they are stress-testing it in an ongoing partnership. The classical critique of Taylorization is acknowledged directly.
• Standardizing steps deskills work and concentrates expert judgment in fewer places, so someone executing only standardized steps may not develop the judgment to notice when the process is failing for a particular construct. The authors reply that a standardized process at least makes the points requiring judgment visible, while conceding the tension is not fully resolved.
• The six criteria can only be judged by whether the process transfers to new constructs and domains, which takes time to establish.

Connected Concepts

• Educational NLP
• Large Language Models (LLMs)
• Assessment Validity
• Educational Measurement
• Research Methods in AIED
• Qualitative Research
• Explainable AI
• Bias Mitigation
• Limitations in AIEd Research
• Pedagogical Partnerships

Connected Articles

• Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education (Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity)
• Comparing GPT and human raters in essay assessment: Variability, bias, and the potential of LLM-based scoring (Comparing GPT and Human Raters in Essay Assessment)
• Ethical implications of prompt injection in AI-mediated grading: An adversarial red-team evaluation (Ethical Implications of Prompt Injection in AI-Mediated Grading)

Citation

Baker, R. S., Mills, C., Lan, A., Lehman, B., & Barany, A. (2026). A Taylorizable Process for Textual Detector Development for Research-Practice Partnerships. Journal of Learning Engineering, 1(1), forthcoming.