AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models

Gayed presents AiAWE, an open-source automated writing evaluation (AWE) system that scores argumentative essays using a LoRA-adapted instruction-tuned large language model (Gemma-3-27B-it). Using a proprietary ETS dataset of 480 TOEFL Independent Writing essays (120 training, 360 evaluation), the system achieves:

These results outperform both LLaMA-3.3-70B and the fine-tuned GPT-3.5 baseline from prior work on the same dataset, demonstrating that open-weight LLMs can match or exceed proprietary fine-tuning for rubric-aligned scoring.

Three broader findings emerge: (1) model scale is not a reliable predictor of downstream performance under LoRA adaptation; (2) identical LoRA hyperparameters produce qualitatively different adaptation behaviors across architectures; and (3) the system runs on a consumer-grade server, making it accessible for institutional deployment. The work contributes to debates about automated assessment quality and the feasibility of Open Source AI tools in Writing Education.

Connected Concepts

Connected Articles

Citation

Gayed, J. M. (2026). AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models. arXiv:2606.12801.