How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures

Synthesis: Compares parallel self-report and objective-based measures of AI literacy built inside the same Concept, Use, Evaluate, Ethics framework for 288 K-12 teachers, and finds the two barely agree. Correlations between the objective and self-reported factors ranged from r = 0.07 to r = 0.24, and latent profile analysis found six profiles: 43 teachers rated themselves consistently high while scoring lower on the objective measure, 59 showed the opposite pattern, and the rest clustered near the mean or split by prior AI literacy experience.

Key Findings

• The study developed and evaluated parallel self-report (SR) and objective-based (OB) measures of teacher AI literacy within a shared Concept, Use, Evaluate, and Ethics framework, enabling direct comparison of perceived versus demonstrated competence.
• Confirmatory factor analyses supported construct validity with good reliability and acceptable fit, but revealed a low correlation between the self-reported and objective-based factors — teachers' perceptions of their AI literacy and their demonstrated performance diverge.
• Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR ≈ OB), and a unique low-SR/low-OB profile concentrated among teachers without prior AI literacy experience.
• The divergence between perceived and demonstrated competence has direct implications for professional development: Self-Assessment alone is an unreliable basis for planning AI training.
• The instruments function as diagnostic tools supporting AI-informed decisions such as growth monitoring and needs profiling, and enable scalable learning-analytics interventions tailored to teacher subgroups.

Study Design & Method

The research responds to the widespread adoption of AI in K-12 education and the resulting need for psychometrically tested measures of teachers' AI literacy. Prior work relied on either self-report or objective-based assessments, with few studies aligning the two within a shared framework. Here, both SR and OB instruments were built on shared dimensions — Concept, Use, Evaluate, and Ethics — so the two measurement approaches could be compared directly rather than studied in isolation. Confirmatory factor analysis established construct validity and reliability, and latent profile analysis grouped teachers by the pattern of agreement or disagreement between their self-reports and objective performance, including how prior AI literacy experience shapes that relationship.

What this means for practice

• Faculty developers. Do not place teachers into AI training from self-report alone. The self-report and objective-based factors were only weakly associated, so what teachers say about their AI skills is not a usable proxy for what they can demonstrate.
• Faculty developers. Administer both instruments before and after training and read them as diagnostics of growth and calibration rather than summarizing them into one AI literacy score.
• Faculty developers. Differentiate support by profile: teachers whose self-ratings outrun their performance (n = 43) need calibration against performance criteria, while the low–low group — found only among teachers without prior AI literacy learning experience (n = 102) — needs foundational experience first.
• Teachers. Cross-check your confidence against a performance task before choosing an AI-focused professional development course; the underestimating profile (n = 59) shows that low self-ratings and near-average performance also occur.
• Researchers. Report self-report and objective measures separately rather than substituting one for the other, and treat prior AI literacy experience as a moderator: it was associated with fewer extreme mismatches, not uniformly higher self-ratings.

Limitations

• Of 358 teachers who completed the survey, 70 did not finish the self-report portion, leaving an analytic sample of 288 — so the self-report and objective analyses rest on a non-random subset of the original respondents.
• The analysis does not separate pre-service from in-service teachers or model years of teaching experience, which the authors identify as a limit on examining how profiles differ across those groups; 46.5% of the analytic sample were pre-service teachers.
• The objective measure was scenario-based but not tailored to subject area or grade level, so it does not capture the subject-specific AI integration that the framework treats as central.
• The authors note that recent additions to AI literacy frameworks (Detect AI, generative AI literacy) are not covered, and that many existing items are overly technical for K–12 educators — the measure may reward familiarity with terminology rather than the capacity to integrate AI into teaching.

Citation

Zhang, S., Xiao, R., Botelho, A. F., Liao, G., Chiu, T. K. F., Stamper, J., & Koedinger, K. R. (2026). How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures.