ChatGPT Solves All Tested Qiskit Homework Assignments

Synthesis: This empirical study by Kaltchenko and Tiwana (Wilfrid Laurier University) tests whether introductory quantum-computing homework can remain autogradable while still forcing students to run, review, and interpret their own results rather than banning AI. Three Qiskit assignment packages were built — seeded basis-state circuits, Quantum Fourier Transform (QFT) with inverse-transform recovery, and seeded Deutsch–Jozsa — each wrapped in deterrence layers (deterministic personalization, non-palindromic bitstrings, varied measurement maps, simulator execution, machine-readable JSON submissions, hidden references, circuit metrics, reflections, optional IBM Quantum execution). For each package a single fixed student-visible instance was tested in 50 separate ChatGPT sessions (150 total); all 150 final artifacts executed and passed the grader, giving each instance zero observed ChatGPT-resiliency under the study's operational definition. The analysis explains why the defenses failed — seeds changed parameters rather than task structure, expected results stayed derivable from visible logic, Scaffolding exposed key steps, and hidden grading verified consistency rather than authorship — and argues that correct artifacts must be complemented by direct assessment of understanding (supervised modification, oral defense, prediction, transfer), a core concern for Academic Integrity and Large Language Models (LLMs) use in Physics Education.

Study design and the threat model

The paper frames the adversary not as a sophisticated attacker but as a minimally engaged student: someone who provides ChatGPT the assignment text, notebook, or student-visible files; requests a complete solution and required artifacts; runs the generated code in Jupyter or Colab; returns an exact installation or runtime error; and submits the finished work. The student never receives the hidden instructor solution, hidden reference functions, expected answers, or grader Feedback. This models direct outsourcing rather than elaborate Prompt Engineering or autonomous agents.

Operational resilience is defined as completion resistance, not plagiarism detection, authorship, or learning: for each package a, observed resiliency is Ra = 1 − Ca/Na, where Na is the number of separate ChatGPT sessions and Ca is the number yielding an executed, grader-accepted completion. A session counted as correct only when the final artifact executed and passed the corresponding grader after ordinary execution and permitted error feedback. Each fixed instance was repeated in 50 separate sessions, holding assignment content constant while resampling the model interaction, so the design tests whether completion was stable rather than a one-off response — not generalization across seeds or variants.

The three assignment packages

Three package types target distinct quantum-programming difficulties. HW1 (seeded basis-state circuits) addresses a frequent Qiskit pitfall — count keys are displayed in classical-register order, which can differ from the order students list qubits — by having students prepare a personalized basis state, apply a seeded bit-flip mask, measure through a direct or reversed qubit-to-classical-bit map, run Aer, explain count-string ordering, and export answers.json. HW2 (QFT followed by inverse-QFT) moves to a recognizable algorithmic workflow where the inverse transform recovers the input (QFT⁻¹QFT|x⟩ = |x⟩), adding circuit depth and operation counts plus an optional hardware extension — but its student-facing notebook is heavily scaffolded. HW3 (seeded Deutsch–Jozsa) requires a seed-specific oracle rather than a textbook circuit: students classify a seeded linear oracle, build the custom oracle and complete circuit, use a personalized measurement map, run Aer, and report metrics and reflections, with an optional hardware run.

Across all three, deterrence layers include deterministic synthetic or student identifiers that regenerate a reproducible configuration, non-palindromic bit patterns, nonzero or custom masks, measurement-map variation, machine-readable JSON schemas, hidden deterministic instructor references (regenerating expected output without a per-student answer key), circuit and transpiler metrics, and reflections tied to individual results. The broader project also explored hidden instructions, canary markers, QPY semantic validation, and noisy-Simulation grading, but these were not mandatory in the 150-session fixed-instance experiment and are not claimed as defeated controls.

Results: all 150 artifacts passed

For each package, Ca = Na = 50, so Ra = 0: 150 of 150 final artifacts executed and passed the corresponding grader, and each fixed instance had zero observed ChatGPT-resiliency. Nine sessions (three per package) were fully archived with transcripts, generated files, logs, and grader outputs under a controlled protocol — each began a new conversation, only student-visible materials were supplied, at most two follow-ups were permitted (limited to an exact runtime error or a request for an omitted required file), and the operator supplied no conceptual hint, independent code fix, expected answer, hidden reference, or grader feedback. Within those nine archived sessions every first-response artifact was already grader-passing, and no archived session required operator code edits or correction of quantum logic.

The only failures were environmental. Eight notebook executions initially stopped at an optional qc.draw("mpl") call because the pylatexenc dependency was missing, and one HW2 grading environment lacked qiskit-aer; each resolved with a single permitted installation or rendering adjustment. These were dependency and Visualization issues, not errors in the quantum algorithm, expected result, output schema, or reflection. Earlier exploratory runs on real IBM hardware (QPE on ibm_marrakesh, QFT and Deutsch–Jozsa on ibm_kingston) showed hardware supplied execution evidence — backend names, job identifiers, transpiled circuits, noisy counts — but did not eliminate the minimally engaged workflow, since ChatGPT generated the code and the operator returned execution data for interpretation.

Why the deterrence layers did not stop ChatGPT

The analysis identifies four reasons the defenses failed. First, personalization changed parameters, not the task class: a deterministic seed prevents every student from receiving the same numeric configuration but does not create a new reasoning problem, because ChatGPT could read the visible generator, specialize the circuit to generated values, and produce a parameterized solution; non-palindromic bitstrings and reversed maps exposed careless implementations but remained explicit, rule-based transformations. Second, scaffolding and predictability weakened execution dependence: in HW2 the notebook exposed the complete QFT and inverse-QFT routines, the expected recovery relationship, the measurement-map calculation, and generic reflection text, so ChatGPT could complete the package without independently deriving the algorithm and could know the ideal count key before running the simulator; HW3 similarly disclosed that a zero mask is constant while a nonzero linear mask is balanced.

Third, hidden grading is not hidden problem information: hidden deterministic references made assignments autogradable and stopped students from reading an instructor answer file, but they did not prevent ChatGPT from deriving a correct answer from the public specification — the graders checked seeded configuration fields, expected bitstrings, ideal counts, dominant outcomes, and reflection presence or length without semantically validating every circuit or establishing independent authorship or understanding. Fourth, machine-readable JSON and reflections created no resistance: LLMs handle structured output well when a schema or example is supplied, and generic or result-specific reflections were generated fluently after ChatGPT had calculated or received the relevant values. The observed minor failures — missing packages, outdated Runtime syntax, occasional bit-order risk — were ordinary debugging issues that a student could paste back into the same conversation, consistent with quantum-code studies showing large gains from execution feedback.

What this means for practice

• Instructors. Add a short oral defense tied to the student's own circuit and counts, or a supervised modification of a qubit mapping, oracle, or input state; all 150 submitted artifacts passed the graders, so a completed notebook establishes nothing about who did the work.
• Instructors. Ask students to predict a new outcome before execution and set a transfer task that changes the algorithmic structure rather than only the seed — the deterrence layers failed because personalization varied parameters, not the task class.
• Assessment designers. Specify which evidentiary claim each artifact supports — that execution occurred, that the circuit is semantically correct, or that the student understands and can transfer — because notebooks, JSON submissions, transpiler metrics, and reflections can all be genuine while assembled through an LLM-directed workflow.
• Instructors. Allow AI for syntax, setup, and debugging and assess the explanation and validation reasoning directly through direct assessment of conceptual understanding; hidden grading verified consistency with the specification, not authorship. Keep the package's personalization, autograding, simulator work and hardware labs — the negative result does not mean abandoning them — and add brief in-class checks that connect code, mathematics and observed hardware noise as a formative counterweight to the take-home artifact.
• Researchers. Report completion resistance as its own quantity: a zero-resilience result measures whether ChatGPT could finish the assignment, not whether a student learned anything.

Limitations

• Three fixed student-visible instances, one per implemented package, were tested at the introductory undergraduate level, 50 ChatGPT sessions each; the design supports a claim about consistency across sessions for those instances, not about other seeds, assignment variants, open-ended design tasks, or semantic circuit validators.
• Complete transcripts, generated versions, logs, and grader output were retained for only 9 of the 150 sessions, so the aggregate success result is verified for all sessions while exact process-level replay is limited to those nine.
• No human participants were studied, so the work measures no student behavior, learning, motivation, or over-reliance, and there was no randomized baseline or ablation isolating the contribution of any single deterrence layer.
• The study answers only for ChatGPT: exact model labels and dates were preserved for the nine archived sessions but not uniformly across the others, and rapid model and Qiskit changes limit temporal reproducibility.

Connected Concepts

• CS Education
• Academic Integrity
• Automated Assessment
• Generative AI
• Assessment
• Large Language Models (LLMs)
• Physics Education
• Formative Assessment
• Prompt Engineering
• Student-AI Interaction

Connected Articles

• Harnessing Generative Artificial Intelligence in Computer Science Education: Pedagogical Innovation, Ethical Responsibility, and the Future of Assessment — Harnessing generative AI in computer science education: pedagogical innovation, ethics, and the future of assessment
• Reshaping Undergraduate Computer Science Education in the Generative AI Era — Reshaping undergraduate computer science education in the generative AI era
• Assessment twins: An approach for strengthening assessment validity in the age of generative AI — Assessment twins for strengthening assessment validity in the age of generative AI
• Teaching Intro AI When the Tools Can Do the Homework: A Course Redesign and a Student Bill of Rights — Teaching intro AI when the tools can do the homework
• Is using artificial intelligence tools for academic work cheating? Student perceptions, ethics, and the impact — Is using AI tools for academic work cheating? Student perceptions and impact on academic performance
• Responsible Assessment in the AI Era: Key Insights from a Future-Focused Conference — Responsible assessment in the AI era

Citation

Kaltchenko, A., & Tiwana, G. (2026). ChatGPT Solves All Tested Qiskit Homework Assignments.