Simulating Students' Java Programming Errors with Large Language Models

This paper investigates whether large language models can serve as scalable proxies for students by simulating realistic logical errors in code submissions. Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, the authors evaluate five LLMs under three prompting strategies: Input-Output (IO), Chain-of-Thought (CoT), and iterative Self-Refine.

Diversity vs. Alignment trade-off: LLMs generate diverse error patterns, but alignment with authentic student errors varies significantly by model. Claude Sonnet 4 achieves the most balanced performance across both dimensions.

Functional indistinguishability: A blinded expert annotation study (N=401) found that synthetic errors are functionally indistinguishable from authentic student errors.

Task difficulty effects: Higher-struggling-level problems elicit more diverse but less student-like errors — LLMs struggle more to simulate realistic mistakes on harder tasks.

Practical implications: Synthetic errors could be integrated into intelligent tutoring systems, teachable agents, and large-scale learning analytics pipelines without waiting for authentic classroom data accumulation.

Methodology

The study used the CodeWorkout dataset with 74,000+ unique student Java submissions. Five LLMs were tested under three prompting strategies. Performance was assessed on two dimensions: diversity (range of distinct error patterns) and alignment (correspondence with authentic student mistakes). A blinded expert annotation study with 401 samples confirmed the indistinguishability of synthetic and authentic errors.

Related Work

This work extends research on LLM-based student simulation and student misconception identification. It connects to programming intelligent tutoring systems and student modeling by offering a scalable method for generating training and evaluation data. The findings also inform AI-generated traces from novice programmers and research on code review with generative AI in CS1.

Connected Concepts

Connected Articles

LLM Student Simulation Misconception Faithfulness

LLM Student Misconception Identification

Programming ITS

AI Generated Traces Novice Programmers

Code Review GenAI Cs1

Citation

Keramati, A., Cao, J., Mohammadi, I., Warschauer, M., & Shi, Y. (2026). Simulating Students' Java Programming Errors with Large Language Models. arXiv:2606.14113.