Development of an AI literacy self-assessment questionnaire in upper primary school students

Synthesis: Thianwan and Srikoon (2025) build and validate the AI Literacy Self-Assessment Questionnaire (AIL-SAQ), a 15-item self-report instrument for students in Grades 4 to 6 (ages 9 to 12) in Khon Kaen Province, Thailand. The scale covers Learning About AI (LAI), Learning About How AI Works (HAI), and Learning for Life with AI (FAI), and is framed explicitly as Self-Assessment of perceived AI Literacy rather than a benchmark of competence. Study 1 (n = 335) used exploratory factor analysis to recover a three-factor structure; Study 2 (n = 579) confirmed it with excellent fit and an overall Cronbach's alpha of .934. The developmental case is the paper's real argument: children at this stage reason about systems and fairness well enough to report on their own emerging understanding, and a validated instrument gives teachers a formative diagnostic they otherwise lack. The authors are also unusually careful about the genre's limit, stating that self-assessment accuracy is constrained by children's developing metacognitive abilities and that scores index perceived rather than demonstrated competence.

Key Findings

• The AIL-SAQ is a 15-item instrument covering Learning About AI, Learning About How AI Works, and Learning for Life with AI, written for Grades 4 to 6 students aged 9 to 12.
• Study 1 (n = 335) recovered a clean three-factor structure; Study 2 (n = 579, evenly split at 193 students per grade) confirmed it in a fresh sample.
• Confirmatory fit was excellent: chi-square = 105.066, df = 87, p = .091, CFI = .994, TLI = .993, RMSEA = .019, SRMR = .022.
• Reliability was high: overall Cronbach's alpha = .934, with subscale alphas of .841 for LAI, .846 for HAI, and .828 for FAI, above the .70 threshold the authors cite.
• Standardized loadings ranged from .638 to .774 and explained variance (R2) from .407 to .599; items i11 and i12 were kept despite lower FAI loadings.
• Study 1 reclassified four items, moving i5 to FAI, i4 to HAI, and i6 and i7 to LAI, so the final model is empirically organized.

Building the AIL-SAQ for upper primary learners

The paper starts from a gap. The authors note that existing AI literacy instruments were designed for older students or professionals, that no standardized assessment framework exists for primary education, and that educators therefore lack agreed benchmarks for what children know about AI. In response they draft 15 items across the three dimensions of the literature: LAI covers foundational knowledge and societal impact, HAI covers mechanisms such as machine learning and data, and FAI covers ethical and applied use of AI in daily life. Crucially, they adopt a perceptual definition of AI literacy, following Casal-Otero et al.: the questionnaire captures students' self-perceived understanding, attitudes, and awareness, not demonstrated skill. Competency-based definitions are contrasted, and the perceptual choice is justified by the instrument's purpose as a formative diagnostic rather than a proficiency test.

What the validation study established

Validation ran in two stages. Study 1 conducted exploratory factor analysis on 335 students using varimax rotation, with a KMO measure of .958, and identified three factors with eigenvalues above 1.0. Study 2 drew 579 new students, evenly distributed across Grades 4, 5, and 6, and ran a first-order confirmatory factor analysis in Mplus. The measurement model was revised first: i5 moved from LAI to FAI, i4 from LAI to HAI, and i6 and i7 from HAI to LAI, following the exploratory loadings. Internal consistency held at .841 for LAI, .846 for HAI, .828 for FAI, and .934 overall. The authors also report inter-factor correlations of .933 to .944 and acknowledge that this near-ceiling association leaves room for a general-factor reading of AI literacy.

What a self-assessment questionnaire can and cannot establish

A self-report instrument establishes what students say about their AI understanding, and this paper is explicit that such a record is not a competence measure. The authors cite developmental constraints: self-assessment accuracy depends on metacognitive ability that is still maturing in children, so a child's estimate of their own skill is a weaker signal than an adult's. They also argue that self-assessment must be taught, supported with guided reflection, teacher feedback, and incentives for accuracy, and that it works best when it targets artifacts or work rather than the self. What the AIL-SAQ does establish is measurement structure for this age group: a stable three-factor model, high internal consistency, and items children can comprehend. What it cannot establish is whether a student can actually explain AI, evaluate an AI-generated claim, or make an independent decision when collaborating with a system.

What this means for practice

• Instructors. Use the AIL-SAQ formatively before, during, and after an AI unit to surface misconceptions and adjust instruction.
• Instructors. Pair it with a task, observation, or artifact review; the instrument records perceived understanding, not demonstrated skill.
• Researchers. Reuse the LAI, HAI, and FAI framing for comparability, and report self-report and objective measures separately.

Limitations

• Both samples came from Khon Kaen Province, Thailand, and no objective or behavioral measure was collected, so reported understanding was never checked against demonstrated skill.
• Inter-factor correlations of .933 to .944 are high enough to support a single general factor; the authors retained three factors on theoretical grounds.
• Test-retest stability is not reported, and four items changed dimension between studies, so the validated model departs from the theoretical draft.

Connected Concepts

• AI Literacy
• Self-Assessment
• Self-Report Measures
• Educational Measurement
• Assessment Validity
• Psychometrically Aware AI
• Student Engagement
• Learners

Connected Articles

• Raising Ethical Awareness of GenAI Use Through Student Self-Assessment in the Transition to Higher Education — Pre- and post-semester self-assessments of ethical GenAI use in a transition-to-university cohort
• How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures — Self-reported and objective measures of teacher AI literacy correlated at only r = 0.07 to 0.24
• Measuring Acceptance of Age-Tiered AI Literacy Guidebooks: A Developmentally Informed Study of K-12 Students and Teachers — Developmentally tiered AI literacy materials for younger learners, with acceptance and validity checks
• The GenAI Skill Bypass: Mapping Divergent Pathways of University Students and Staff AI Literacy — An inverted skill profile found in 158 GenAI literacy self-assessments
• The Generative Artificial Intelligence Literacy Scale (GAILS): Development, Validation, and Measurement Invariance Across Sex and Occupational Status Groups — Scale development and validation with measurement invariance testing across groups
• The Absent Cognitive Baseline: Theorizing a Structural Gap in AI-Native College Students' Academic Self-Assessment — What academic self-assessment can and cannot rest on when the experiential record is missing

Citation

Thianwan, K., & Srikoon, S. (2025). Development of an AI literacy self-assessment questionnaire in upper primary school students. Social Sciences & Humanities Open.