Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot

Synthesis: A custom Socratic AI chatbot deployed in a large-enrollment introductory mechanics course with 150 first-year STEM majors, demonstrating that AI-driven Socratic dialogue can foster expert-like reasoning while generating fine-grained learning analytics for physics education research.

Key Findings

1. Socratic dialogue improves question specificity. Student question specificity rose dramatically from ~10–15% on the first conversational turn to 100% on the final turn, indicating that sustained Socratic interaction trains students to formulate increasingly precise, expert-like physics questions.
2. Specificity correlates with academic performance. Self-reported expected course grade showed a significant positive correlation with question specificity (Pearson r = 0.43), suggesting that the ability to formulate precise physics questions — a skill the chatbot explicitly cultivates — is linked to broader course outcomes.
3. Students rated the chatbot positively on knowledge-building. Post-interaction surveys yielded a median rating of 4.0/5 for knowledge-based skills and 3.4/5 for overall effectiveness, indicating acceptable student reception for a tool deployed at scale.
4. Dual-purpose design enables both instruction and research. The chatbot served simultaneously as a Socratic Method teaching tool and as a data-collection instrument for Learning Analytics, with full dialogue transcripts enabling fine-grained analysis of student reasoning patterns.

What this means for practice

• Instructors. Run Socratic AI dialogue even in large courses: with 150 first-year STEM majors, the share of students asking specific physics questions rose from about 10–15% on the first turn to 100% by the final turn.
• Instructors. Read question specificity as a progress signal, since it correlated with self-reported expected course grade (r = 0.43), and students still asking broad questions late in a session rated the tool lower.
• Instructors. Anchor each session to one context-rich problem and let students work individually with the chatbot before debriefing the physics reasoning together.
• Researchers. Log complete transcripts: the same deployment doubles as a research instrument, capturing how students formulate problems rather than only their final answers.
• Instructors. Keep the chatbot complementary to instruction — the authors report uneven motivation gains and warn against treating it as a replacement for instructors or peer collaboration.

Limitations

• The study ran in a single course with 150 first-year STEM majors using one problem scenario, the human-cannonball spring launch, which the authors state may not generalize.
• Outcome evidence is self-reported expected course grade and post-activity survey ratings; no post-test, control group, or retention tracking was included.
• Question specificity was coded with a coarse binary broad/specific distinction that the authors say overlooks conceptual, procedural, and verification nuances.
• The fixed question progression and GPT-4o hints limit adaptivity, and only fully completed surveys entered the analysis.

Citation

Hashmi, A., et al. (2025). Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot. v1.