Chat Debugging: An Exploratory Study of Human-AI Collaboration to Debug Analog Circuits

Synthesis: This exploratory study investigates how undergraduates use LLMs to debug malfunctioning analog circuits under exam conditions, identifying both promising collaborative patterns and critical limitations. Through thematic analysis of student chat logs, the authors find that off-the-shelf LLMs offer considerable domain knowledge and sensible debugging suggestions, yet struggle with 2D/3D image-based reasoning and display unjustified confidence. Students, in turn, show deficits in fundamental concepts and critical thinking during AI-assisted debugging.

Study Design

• Context: Undergraduate students debugging predetermined buggy analog circuits on breadboards and PCBs
• Conditions: Lab final exam in ECEN 3314 (Electronic Devices and Applications) at Oklahoma State University, Spring and Fall 2025, under 30-minute time pressure
• Data: 17 voluntarily shared chat logs from student-LLM conversations plus written answer sheets
• Analysis: Inductive thematic analysis of interaction patterns at both technology and psychological levels

Key Findings

1. Students overwhelmingly preferred image-based debugging, uploading photos of circuits and exam questions to capture full context — a pattern that grew from 2/5 chat logs in Spring 2025 to 5/12 in Fall 2025.
2. Without any domain-specific fine-tuning, off-the-shelf LLMs were able to suggest true root causes among their zero-shot debugging recommendations, at times demonstrating chain-of-thought reasoning.
3. LLMs' 3D spatial and visual reasoning was unreliable: nearly all factual errors followed image inputs, as models could not correctly infer wiring and structural relationships from breadboard or PCB photos.
4. LLMs displayed unjustified confidence in wrong diagnoses, while students showed deficits in fundamental circuit concepts and offloaded critical thinking to the AI.

Background: The Case for Human-AI Debugging Collaboration

Debugging is a high-stakes, undertaught skill in hardware engineering. In the semiconductor industry, post-silicon debugging is notorious as the "Schedule Killer" — a hardware engineering task that can consume a third to half of a project's time and millions in fabrication costs, yet few universities offer dedicated debugging curricula. The paper grounds its argument in the cognitive task analysis of troubleshooting: humans generate few hypotheses per problem, suffer from working-memory overload during problem-space reduction, and lean heavily on experience that novices simply lack.

Against these pain points, the authors position LLM collaboration as a potential game-changer. LLMs trained on massive corpora can act as a form of "virtual experience," offer personalized emotional support through immediate chatbot feedback, and — because analog circuits lack a universal hardware description language — let students describe circuits in natural language without loss of information. The aim is not to replace the engineer but to combine human observation with AI reasoning, planning, and pattern recognition in a genuinely collaborative debugging loop.

Research Questions & Methodology

The study poses three research questions: (1) What are students' usage patterns when debugging with LLMs? (2) What strengths do LLMs show in guiding debugging? (3) What gaps exist in LLM technologies or student skills? To answer them, the authors analyzed chat logs from a timed 30-minute lab final exam in which students debugged randomly assigned buggy circuits on breadboards and PCBs, shared their full chat logs afterward, and completed a written worksheet on symptoms, root causes, and fixes. Analysis followed an inductive thematic analysis capturing both factual correctness and underlying trust in AI suggestions.

Findings: Students' Usage Patterns

Theme 1 — Students used images to capture all context. Contrary to the expectation of verbal descriptions, students routinely photographed both the physical circuit and the exam assignment and uploaded them to the LLM. The authors attribute this to time pressure and the convenience of photos. This Multimodal AI preference sets up the central tension of the study: students want to lean on visual evidence, but the AI's visual reasoning is precisely where it fails.

Findings: LLM Strengths

Theme 2 — LLMs suggested true root causes at zero-shot. In a representative exchange, ChatGPT listed four potential issues for a common-emitter amplifier and its fourth hypothesis pointed directly to the actual bug (improper biasing). The model then demonstrated chain-of-thought reasoning, correctly locating the transistor package (TO-92) and reasoning about the relative emitter/base/collector positions. Notably, the student still failed to finish in time — evidence that a good suggestion alone does not guarantee successful debugging.

Findings: LLM and Student Gaps

Theme 3.1 — Image reasoning is the weak point. Almost all factual errors followed image inputs. ChatGPT misjudged a ground clip's rail connection and misread a resistor's insertion point because it could not discern three-dimensional spatial relationships from a single top-view photo.

Theme 3.2 — The tone of confidence is concerning. After a major visual recognition error, ChatGPT asserted it was "99% confident" about wrong op-amp pins and offered to "mark exactly where each connection should go." The calibration of confidence in multimodal contexts is a genuine risk: confident-sounding but wrong diagnoses can actively mislead students who lack the knowledge to verify them.

Theme 3.3 — Students lacked fundamentals and offloaded critical thinking. Students asked what a common-source amplifier's expected output is, whether an amplified BJT signal is AC or DC, and confused an oscilloscope probe setting with the Volt/div knob — relying on the AI for circuit fundamentals and lab skills. This reflects a pattern of cognitive offloading in which students delegate verification and conceptual judgment to the model rather than exercising critical thinking over its outputs.

What this means for practice

• Instructors. Tell students up front that a confident reading of a breadboard or PCB photo is the least reliable part of the model: nearly all factual errors in this study followed image inputs, so a plausible description of wiring is not evidence.
• Instructors. Keep fundamentals in the room and out of the model's hands — students asked the AI for basics they were expected to hold themselves and confused an oscilloscope probe setting with the Volt/div knob.
• Instructors. Require the fix, not the suggestion: ChatGPT named the correct bug zero-shot in one exchange and the student still ran out of time, so grade the verification step and the repair rather than the chat.
• Designers. Do not make image-based debugging the primary path in a course workflow while 3D spatial reasoning remains unreliable; invest instead in domain-specific or 3D visual tooling.
• Instructors. Treat these patterns as hypotheses to test in your own lab rather than settled guidance, since the themes rest on 17 voluntarily shared logs and the authors' planned mixed-methods follow-up is not yet in: limitations in AI-education research.

Limitations

• Exploratory pilot on a small, self-selected sample. The study rests on 17 chat logs that students volunteered to share, so its themes cannot be generalized beyond this setting.
• Single-method design. The evidence is qualitative only; the authors' comprehensive mixed-methods approach, pairing the themes with quantitative measures, is planned future work rather than part of this study.
• Conclusions remain provisional. More data collection is needed before the themes can be made robust (see limitations in AI-education research).

Citation

Hu, J., & Ash, A. (2026). Chat Debugging: An Exploratory Study of Human-AI Collaboration to Debug Analog Circuits. v1.