AI Ed Wiki logoAI Ed Wiki
Use with AI

📄 Research Article

AI-mediated authentic assessment and metacognitive reflection: A mixed-methods study of the AAIWA model

● high · 2026-08-16 · authentic-assessment, feedback, metacognition, teacher-education, pre-service-teachers, academic-writing

Synthesis: Oktoma, Nugraha and Hidayat (2026) introduce the Authentic AI-Assisted Writing Assessment (AAIWA) model, integrating authentic rubric-based assessment, condition-responsive AI Feedback Quality, and structured Metacognition reflection as a unified recursive architecture within Indonesian Teacher Education education. Existing models treat these components additively, creating a design gap that is especially acute in underrepresented Global South contexts. AAIWA computationally embeds rubric criteria within the Feedback engine and positions structured reflection as a dynamic input that recalibrates subsequent feedback cycles, rather than a post-hoc add-on. The prototype is a constrained hybrid system: a rule-based NLP layer governs all pedagogically consequential decisions while GPT-3.5-turbo serves exclusively as a language-articulation module, so "condition-responsive" feedback denotes rule-conditioned responsiveness, not autonomous LLM reasoning. In a mixed-methods R&D design with 75 pre-service teachers across three Indonesian universities across four iterative writing cycles, academic writing improved substantially (d = 0.92) and metacognitive reflection rose (d = 0.98), with a moderate writing–reflection correlation (r = .54) offering partial mechanistic support for the integrated design.

Key Findings

Integrated recursive architecture. Unlike additive "AWE + reflection" designs, AAIWA is genuinely recursive: rubric-embedded evaluation, condition-responsive iterative feedback, and reflection-driven recalibration interact across drafting cycles, with each cycle's reflection output adjusting feedback priorities for the next. Three contributions are claimed: the first architecturally integrated model unifying authentic rubrics, condition-responsive AI feedback and metacognitive reflection; preliminary feasibility evidence in a Global South teacher-education context; and a validated instrumentation suite (Authentic Writing Rubric, AWR; Metacognitive Reflection Inventory, MRI).

Constrained hybrid system. A rule-based NLP layer governs all pedagogically consequential decisions (feedback gating, threshold branching, error-type routing), while GPT-3.5-turbo acts only as a language-articulation module. Four condition-responsive dimensions operationalise this: (a) performance level — AWR sub-score threshold branching (≤2.0 → directive Scaffolding + models; 2.1–3.0 → diagnostic explanation + revision cues; >3.0 → higher-order rhetorical prompts); (b) error type — Transformer-classified category (argument, evidence, coherence, lexis, grammar) activates category-specific templates; (c) rhetorical structure — macro-structure feedback precedes sentence-level correction; and (d) metacognitive stage — MRI reflection depth routes between structured prompts (low depth) and evaluative prompts (high depth).

Writing gains. Overall AWR total rose from 43.2 to 50.3 (+16.5%, Cohen's d = 0.92, 95% CI [0.65, 1.19], p < .001). All six dimensions improved with large effects: Reflective Depth (D6) +37.7% (d = 0.91), Argument Quality (D1) +33.3% (d = 0.87), Use of Evidence (D2) +32.6% (d = 0.85), Academic Style (D5) +30.7% (d = 0.83), Organization (D3) +30.3% (d = 0.82), and Coherence (D4) +31.4% (d = 0.79). The two largest effects correspond to competencies most explicitly targeted by the rubric-embedded feedback architecture and reflection cycle; a one-way ANOVA showed no between-institution difference in gains (F(2,72) = 0.18, p = .84).

Metacognitive reflection gains. MRI mean item score rose from 2.31 to 2.78 (+20.7%, d = 0.98, 95% CI [0.69, 1.27], p < .001). Planning Strategies (Section A, d = 0.84) and Evaluation Skills (Section C, d = 0.83) improved markedly, while Monitoring Processes (Section B, d = 0.43) remained medium — suggesting differential metacognitive development. A moderate writing–reflection gain correlation (r = .54, 95% CI [.36, .68]) offers partial mechanistic support for the integrated design.

Validation. Expert validation yielded strong content validity: AWR CVI = 0.84 and MRI CVI = 0.82 (both above the 0.78 threshold), Cronbach's α = 0.84 for MRI, and ICC(2,1) = .87 for rater-mediated evaluation. Two independent coders achieved κ = 0.82 on qualitative themes.

Qualitative themes. Thematic analysis (n = 15 interviews) surfaced three themes: (1) enhanced metacognitive awareness, especially among developing-band students; (2) motivational enhancement through immediate, non-judgmental feedback (alongside concerns about occasional AI–instructor discrepancies); and (3) progressive internalisation of structured reflection as an integrated learning process rather than an added assessment requirement.

Design context and implication. Three structural conditions define the Indonesian implementation environment: infrastructural disparities constraining cloud-based AI, a teacher-centred pedagogical tradition where student self-assessment is undervalued, and a Language Learning context requiring second/third-language cognitive processing. The authors caution the findings are associated with, not causally attributable to, AAIWA (no randomised control group), and call for controlled replication. The model offers a design for Authentic Assessment with AI in Teacher Education that keeps pedagogically critical decisions in a transparent rule-based layer while using LLMs for language articulation.

Connected Concepts

Connected Articles

Citation

Oktoma, E., Nugraha, D., & Hidayat, A. (2026). AI-mediated authentic assessment and metacognitive reflection: A mixed-methods study of the AAIWA model. Computers and Education Open, 100395. https://doi.org/10.1016/j.caeo.2026.100395