AIEd Knowledge BaseAIEd KB
On this page

Research Article

Design and Evaluation of an AI-Supported Automated Programming Assessment Environment for Concept-Aligned Reinforcement in Engineering and Computing Education

created 2026-09-20 · updated 2026-09-20

Synthesis: Lan (2026) evaluates an AI-supported automated programming assessment environment that turns verified incorrect submissions into governed, concept-aligned micro-assessments delivered through a mobile chatbot. Rather than using Generative AI as an open-ended tutor or grading agent, the environment makes it a constrained orchestration layer inside a deterministic workflow: a rule-based engine alone executes code and decides correctness, and AI runs only after a learner clicks a misconception clarification prompt. In a quasi-experimental pretest–posttest study with 90 first-year undergraduates in Taiwan (45 experimental, 45 control), the experimental group reached higher post-intervention concept-level mastery across all five programming components (M = 0.688 versus 0.567; Cohen's d = 1.05) and reported greater Self-Efficacy gains and higher post-intervention Motivation. System logs showed 51.3 completed micro-assessment interactions on average, 46% of them outside class sessions. The author reads the results as evidence for the integrated Formative Assessment environment rather than for isolated effects of generative AI or extra practice.

Key Findings

  1. Each verified incorrect submission linked to a governed, single-concept micro-assessment: a deterministic engine alone graded code, while AI generated diagnostics only after learners requested clarification.
  2. In the 5-week, five-module intervention, 90 first-year undergraduates were split into 45 experimental and 45 control students, both completing the same 50 macro-level Python tasks.
  3. Post-intervention overall concept-level mastery was 0.688 (SD = 0.110) experimental versus 0.567 (SD = 0.119) control, t(88) = 4.98, p < 0.001, Cohen's d = 1.05.
  4. ANCOVA adjusting for baseline mastery kept the advantage significant, F(1, 87) = 25.43, p < 0.001, partial η2 = 0.226, with adjusted means of 0.689 and 0.566.
  5. Logs showed sustained, low-friction engagement: 51.3 completed micro-assessment interactions on average (SD = 9.5), 46% outside class, and 44.2 s mean response latency (SD = 8.5).
  6. Within that group, completed micro-assessments (β = 0.42), off-class participation (β = 0.36), and shorter latency (β = −0.29) were associated with mastery, explaining 47% of its variance.
  7. Self-efficacy gains were larger for the experimental group (1.11 versus 0.42; d = 0.68) and post-intervention motivation was higher (3.76 versus 3.16; d = 0.81).

Keeping grading deterministic while AI supports learning

The architecture separates two layers. A deterministic automated assessment engine delivers instructor-authored macro-level tasks, executes submissions, and decides correctness through rule-based input–output matching; AI does not participate in code execution, grading, or correctness determination. Only after the engine verifies an incorrect submission and the learner clicks a "Misconception clarification prompt" does the reinforcement layer run, and it never revises the verdict. Stage 1 uses gpt-4o-mini at temperature 0.3 to select one focal knowledge component from a predefined whitelist and emit a structured JSON handoff; Stage 2 uses gpt-4.1-mini at temperature 0.3 to generate one learner-facing, four-option item from that handoff. The design turns verified failure evidence into a traceable diagnostic record while preserving the platform verdict, keeping Guardrails and instructor oversight around the Large Language Models (LLMs).

Learner-initiated reinforcement and how learners used it

Micro-assessments were assessment-triggered but learner-initiated, so learners did not receive a fixed sequence of items; delivered items reflected the concepts implicated in their own failed submissions. A hybrid allocation strategy reused validated items when a matching concept context already existed and otherwise generated new ones, producing 385 system-level unique items. Two access pathways existed: the post-failure clarification prompt and learner-initiated review modes (Personal Learning, Set-Based Learning, Comprehensive Challenge). System logs showed 51.3 completed interactions on average, 46% occurring outside class time, and a 44.2 s mean response latency, with off-class participation mainly tied to Comprehensive Challenge. Within-condition regression associated these indicators with mastery — read as process-level signals, not causal mechanisms, since engagement may also reflect self-regulation or help-seeking tendency.

Affective outcomes and expert validation of generated items

Self-efficacy rose from 2.76 to 3.87 in the experimental group versus 2.82 to 3.24 in the control group (gain difference p = 0.002, d = 0.68), and post-intervention motivation was higher (3.76 versus 3.16, p < 0.001, d = 0.81). The author attributes the self-efficacy advantage to the combined influence of concept-aligned reinforcement, extra practice, and repeated verification-first Feedback rather than conceptual learning alone, and notes motivation was measured only at posttest. Internal consistency was high across both instruments and timepoints. Item quality was checked twice: prompt-level constraints, concept-tag consistency checks, and runtime screening for near-duplicates and solution leakage; plus human review of 116 sampled Stage 1–Stage 2 pairs (roughly 30% of the 385 unique items) by two programming education experts, whose Cohen's kappa values ranged from 0.68 to 0.85.

What this means for practice

  • Instructors. Turn verified incorrect submissions into triggers for optional single-concept reinforcement: the experimental group completed 51.3 micro-assessments, 46% outside class, reaching 0.688 mastery versus 0.567.
  • Software developers. Confine AI to a governed support layer: constrain prompts to one whitelisted concept, require schema-valid JSON, and enforce concept-tag consistency between handoff and item.
  • Assessment professionals. Log every handoff–item pair and budget expert review: 116 sampled pairs (about 30% of 385 unique items) reached Cohen's kappa of 0.68–0.85.

Limitations

  • The workflow is model-version-specific: Stage 1 used gpt-4o-mini and Stage 2 gpt-4.1-mini at temperature 0.3, with no re-validation across later models reported.
  • The quasi-experimental intact-class design limits causal claims; with no time-on-task-matched control, components cannot be separated.
  • Evidence is short-term and single-site: 90 students in one 5-week Python course in Taiwan, with no difficulty calibration, delayed posttest, or trust data.

Connected Concepts

Connected Articles

Citation

Lan, Y.-F. (2026). Design and Evaluation of an AI-Supported Automated Programming Assessment Environment for Concept-Aligned Reinforcement in Engineering and Computing Education. Computer Applications in Engineering Education, 34, e70271.

Metadata

Study designExperiment, System Development
DisciplineCs Education
LevelHigher Ed
Intended audienceInstructors, Software Developers
Learning and instructionMastery Learning, Misconceptions about AI, Motivation, Self-Efficacy, Self-Regulated Learning
Technologies and techniquesConversational AI, Generative AI, Learning Analytics, Large Language Models (LLMs)
Assessment and measurementAutomated Assessment, Feedback, Formative Assessment, Learning Gains
Equity, ethics, and responsible useAI Use and Disclosure Statements, Guardrails
SourceHistoryEditChat

Embed this page

Copy the code below to embed a chromeless version of this page in a learning management system or other website. The embedded view hides the site header, navigation, and footer.