Integrating AI Into Computational Thinking: Development and Validation of an Assessment Tool for Higher Education Students

Synthesis: This study develops and validates the Computational Thinking in AI Training Test (CTAT) — a 34-item multiple-choice instrument built with the Evidence-Centered Design framework and validated via expert review, cognitive interviews, and a field test with 461 higher-education students (analyzed with IRT). CTAT shows robust psychometric properties for assessing Computational Thinking within AI-training contexts, and reveals that students struggle most with data representation, logical-operator sequencing, and loop structures.

Key Findings

1. A valid and reliable CT assessment for AI contexts. CTAT (34 MC items) was developed via Evidence-Centered Design, refined through expert review and cognitive interviews, and field-tested with 461 Year 1–2 students from two vocational schools and two colleges in southern China. IRT analysis confirmed robust psychometric properties.
2. CT skills improve across educational pathways. Students' computational thinking improved as they progressed, with no significant gender-based differences observed.
3. Common student difficulties. Students tended to struggle with identifying appropriate data representation, applying logical operators in correct sequence, and differentiating loop structures — actionable insights for teaching text-based programming.
4. Addresses a scarcity gap. Integration of AI principles into CT education and assessment remains scarce; CTAT directly addresses assessing CT within AI-training programs at the higher-education level.

What this means for practice

• Instructors. Target the three item families students failed most — identifying appropriate data representation, applying logical operators in the correct sequence, and differentiating loop structures — with explicit worked examples, rather than distributing time evenly across the introductory text-based programming syllabus.
• Instructors. Use the diagnostic profile, not just the total score: CT item difficulty rose sharply on items needing complex logical reasoning (e.g., Item 13, b = 1.566), so reteach the specific concepts that response patterns expose.
• Designers. Pair a multiple-choice conceptual instrument with project-based tasks when assessing CT in AI contexts: the format measures conceptual understanding reliably but cannot evidence students' competence in creating AI models.
• Researchers. Replicate the item-difficulty findings with learners outside CS and AI majors, since all 461 field-test participants had completed or were completing AI fundamentals or Python programming courses.

Limitations

• The field test drew 461 Year 1–2 students from two public colleges and two vocational schools in China, all majoring in CS or AI and all having taken or taking AI fundamentals or Python courses, so the item-difficulty findings may not generalize to other disciplines or to students without programming experience.
• The instrument currently exists only in English and simplified Chinese, which limits administration and replication in other language contexts.
• The sample came from CS- and AI-related fields where male students are dominant, so the null gender finding rests on an imbalanced sample rather than a well-powered comparison.
• The test design followed Brennan and Resnick's framework, which covers CT concepts and practices but not CT perspectives, leaving one dimension of CT unmeasured.

Connected Concepts

• Computational Thinking
• Assessment
• Assessment Validity
• Item Response Theory
• CS Education
• Generative AI
• Higher Education

Connected Articles

• Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work — LLM cognitive diagnosis in handwritten math
• Automated Grading of Linux/Bash Examinations Using Large Language Models — Automated grading of Linux Bash exams

Citation

Zhang, S., & Zhang, S. (2026). Integrating AI into computational thinking: development and validation of an assessment tool for higher education students. International Journal of STEM Education, 13, 49.