Arthur: An artificial intelligence powered teaching assistant system for Engineering Economics class

Synthesis: Arthur: An artificial intelligence powered teaching assistant system for Engineering Economics class — Yin, Karakaya, Bass, and Cai (2026) introduce Arthur, an AI-powered teaching assistant that delivers real-time, personalized Feedback on Calculated Formula Questions (CFQs) in an undergraduate Engineering Economics course, a domain where handwritten, unstructured solutions have previously blocked AI support. The authors propose a full life-cycle framework spanning data curation, data augmentation, model training, and student-facing deployment, training an XGBoost "solution diagnosis backbone" per CFQ to predict likely mistakes from students' submitted numerical answers. A dialogue-based interaction scheme and question-bank web interface operationalize the system to balance feedback accuracy against collection efficiency within higher education engineering courses.

Key Findings

• A full life-cycle framework for unstructured CFQs. The study's core novelty is a comprehensive pipeline—hand-written data curation from previously graded graded submissions, random-masking data augmentation, AI backbone training, and student-facing interface deployment—providing a scalable pathway for ITS in engineering courses where pen-and-paper solutions lack structured digital data, unlike multiple-choice, coding, or essay assignments.

• Three data barriers addressed. Existing AI tutoring feedback work was constrained to assignments with readily available structured data. For open-ended CFQs, the authors overcome (1) rare digitalization of handwritten solutions, (2) multiple valid solution approaches that complicate diagnosis evidence, and (3) small-scale, imbalanced datasets drawn only from incorrect submissions.

• Random masking for data augmentation. Adapting an NLP-style masking idea (Devlin et al., 2019), the authors generate augmented training copies by masking input features to "NaN" at varying probabilities α (0.1–0.9), preserving the logical dependencies of tabular solution data while simulating students who provide partial answers—outperforming interpolation approaches like SMOTE that would violate CFQ structure.

• XGBoost solution diagnosis backbones. Each CFQ gets its own dedicated multi-label classification backbone mapping intermediate and final numerical answers (X) to instructor rubric mistake labels (Y). The model diagnoses mistakes from submitted numbers alone, bypassing access to full written solutions, and achieves an average precision of 0.81, recall of 0.79, and accuracy of 0.65.

• Intermediate answers and augmentation improve diagnosis. Ablation experiments (five-fold CV, paired t-tests) show that training and testing with both intermediate and final answers yields the best performance, and that random-masking augmentation significantly improves results over full-features-only training. Training on only final answers gives good but not best results and is more straightforward to collect.

• Dialogue-based interaction to balance accuracy and efficiency. Rather than forcing full answers (which can constrain students' problem-solving), Arthur starts by requesting only the final answer, then iteratively prompts for specific intermediate answers when the backbone's prediction confidence falls below a 0.8 threshold—adaptively reducing false positives and the burden on student input.

• Operationalization and generalizability. The system was deployed as a web interface with a curated question bank covering ten Engineering Economics topics, integrating rubric labels into natural-language feedback templates. The framework is designed to generalize to CFQs in other engineering disciplines, with discussion of representation and interaction biases, human annotation error, and opportunities for LLMs to automate handwriting recognition.

What this means for practice

• Instructors. Start each feedback dialogue with the final answer and request intermediate answers only when the model's confidence falls below the 0.8 threshold, which preserves students' own solution approaches while reducing false positives.
• EdTech designers. Augment small, imbalanced rubric datasets with random masking rather than interpolation such as SMOTE, which would violate the logical dependencies of calculated-formula solutions while masking at probabilities from 0.1 to 0.9 preserves them.
• Instructors. Keep collecting intermediate answers during grading: models trained and tested on intermediate plus final answers outperformed those using final answers alone, which matters most for the less frequent mistakes.
• EdTech designers. Budget for per-question models and course expertise — each calculated formula question needs its own XGBoost backbone and a careful identification of critical intermediate answers before deployment.
• Instructors. Disclose the system's limits inside the course, as this deployment did by informing students of Arthur's limitations, enabling feedback uploads to improve the system, and offering office hours for clarification.

Limitations

• Evidence comes from a single undergraduate Engineering Economics course and ten calculated formula questions, with training data hand-curated from previous semesters' graded handwritten submissions — a small, imbalanced dataset that random masking was used to enlarge artificially.
• Prediction quality is modest: precision 0.81 and recall 0.79 but accuracy 0.65, so roughly a third of exact-match mistake diagnoses are wrong, which is why the authors had the system inform students of its own limitations.
• No learning outcome was measured: the study reports model performance and a deployed question-bank website, not student performance with the feedback.
• Course-specific expertise is still required to choose questions and critical intermediate answers, which the authors name first among their three limitations because it restricts how far the question bank can expand; they also flag representation bias from imbalanced mistake frequency and interaction bias as risks.

Connected Concepts

• Intelligent Tutoring
• Formative Assessment
• Feedback
• Machine Learning
• Higher Education
• Engineering Education

Connected Articles

• Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models — LLM-based error diagnosis and personalized feedback for tutors on structured problems
• Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most — diagnosing feedback gaps in AI tutoring
• Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice Experiment — productive AI tutoring for mastery learning in mathematics
• LLM-generated formative feedback in education: A qualitative systematic literature review — systematic evidence on LLM-generated formative feedback
• OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research — an open-source adaptive tutoring platform

Citation

Yin, Z., Karakaya, E., Bass, K., & Cai, H. (2026). Arthur: An artificial intelligence powered teaching assistant system for Engineering Economics class. International Journal of Artificial Intelligence in Education, 36, Article 100003.