Keramati, Cao, Mohammadi, Warschauer & Shi (2026) — University of California, Irvine. 📄 Full text (arXiv)
Summary
This paper investigates whether large language models can serve as scalable proxies for students by simulating realistic logical errors in code submissions. Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, the authors evaluate five LLMs under three prompting strategies: Input-Output (IO), Chain-of-Thought (CoT), and iterative Self-Refine.
Key Findings
- Diversity vs. Alignment trade-off: LLMs generate diverse error patterns, but alignment with authentic student errors varies significantly by model. Claude Sonnet 4 achieves the most balanced performance across both dimensions.
- Functional indistinguishability: A blinded expert annotation study (N=401) found that synthetic errors are functionally indistinguishable from authentic student errors.
- Task difficulty effects: Higher-struggling-level problems elicit more diverse but less student-like errors — LLMs struggle more to simulate realistic mistakes on harder tasks.
- Practical implications: Synthetic errors could be integrated into intelligent tutoring systems, teachable agents, and large-scale learning analytics pipelines without waiting for authentic classroom data accumulation.
Methodology
The study used the CodeWorkout dataset with 74,000+ unique student Java submissions. Five LLMs were tested under three prompting strategies. Performance was assessed on two dimensions: diversity (range of distinct error patterns) and alignment (correspondence with authentic student mistakes). A blinded expert annotation study with 401 samples confirmed the indistinguishability of synthetic and authentic errors.
Related Work
This work extends research on LLM-based student simulation and student misconception identification. It connects to programming intelligent tutoring systems and student modeling by offering a scalable method for generating training and evaluation data. The findings also inform AI-generated traces from novice programmers and research on code review with generative AI in CS1.
Related Pages
- llm-student-simulation-misconception-faithfulness — Related work on LLM faithfulness in simulating student misconceptions
- llm-student-misconception-identification — Complementary work on identifying student misconceptions with LLMs
- programming-its — Programming-focused intelligent tutoring systems that could use synthetic errors
- student-modeling — Student modeling approaches that could benefit from generated error data
- llm-feedback-programming-classroom — LLM feedback in programming education contexts
- learning-analytics — Analytics pipelines that could incorporate synthetic student data
Citation
APA: Keramati, A., Cao, J., Mohammadi, I., Warschauer, M., & Shi, Y. (2026). Simulating Students' Java Programming Errors with Large Language Models. arXiv:2606.14113.