---
source_url: https://arxiv.org/abs/2604.03926
ingested: 2026-05-07
sha256: 69882f295b179f3e1a85d65c4e709934be1c2bbe0e8270ba0c3aba2024632ed8
---
# CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation

**Authors:** Xiaojing Duan, Frederick Nwanganga, Chaoli Wang  
**arXiv:** 2604.03926  
**Venue:** Full paper accepted as short paper at 27th International Conference on Artificial Intelligence in Education (AIED 2026)  
**Submitted:** 5 Apr 2026

## System Overview

CODE-GEN is a human-in-the-loop, retrieval-augmented generation (RAG)-based agentic AI system designed to generate context-aligned multiple-choice questions that develop student code reasoning and comprehension abilities.

### Architecture
| Component | Function |
|-----------|----------|
| **Generator Agent** | Produces multiple-choice coding comprehension questions aligned with course-specific learning objectives |
| **Validator Agent** | Independently assesses content quality across seven pedagogical dimensions |
| **Specialized Tools** | Augment both agents to enhance computational accuracy and verify code outputs |

## Evaluation Study

- 6 human subject-matter experts (SMEs)
- 288 AI-generated questions judged
- 2,016 human-AI rating pairs
- 131 instances of qualitative feedback

Human-validated success rates range from 79.9% to 98.6% across the seven pedagogical dimensions.

## Key Findings

### High Reliability (AI-Strong Dimensions)
CODE-GEN achieves high reliability on dimensions suited to computational verification and explicit criteria matching:
- Question clarity
- Code validity
- Concept alignment
- Correct answer validity

### Human Expertise Required
Human judgment remains essential for dimensions requiring deeper instructional judgment:
- Designing pedagogically meaningful distractors
- Providing high-quality feedback that reinforces understanding

### Strategic Implication
These findings inform the strategic allocation of human and AI effort in AI-assisted educational content generation.