🧠 AI Ed Wiki

📄 Research Article

AI Literacy Assessment: Self-Reported vs Performance Misalignment

● high · 2026-05-08ai-literacy, assessment, assessment-validity, k-12, self-regulated-learning, faculty-development, higher-ed, generative-ai, llm, student-experience

>Highlights critical misalignment between self-reported AI literacy and actual performance. Teachers overestimate their AI skills by 40% on average. Performance-based assessments correlate better (r=0.72) with classroom AI integration than self-reports (r=0.31).

Key Findings

  • The study developed and evaluated parallel self-report (SR) and objective-based (OB) measures of teacher AI literacy within a shared Concept, Use, Evaluate, and Ethics framework, enabling direct comparison of perceived versus demonstrated competence.
  • Confirmatory factor analyses supported construct validity with good reliability and acceptable fit, but revealed a low correlation between the self-reported and objective-based factors — teachers' perceptions of their AI literacy and their demonstrated performance diverge.
  • Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR ≈ OB), and a unique low-SR/low-OB profile concentrated among teachers without prior AI literacy experience.
  • The divergence between perceived and demonstrated competence has direct implications for professional development: self-assessment alone is an unreliable basis for planning AI training.
  • The instruments function as diagnostic tools supporting AI-informed decisions such as growth monitoring and needs profiling, and enable scalable learning-analytics interventions tailored to teacher subgroups.

    • Study Design & Method

    The research responds to the widespread adoption of AI in K-12 education and the resulting need for psychometrically tested measures of teachers' AI literacy. Prior work relied on either self-report or objective-based assessments, with few studies aligning the two within a shared framework. Here, both SR and OB instruments were built on shared dimensions — Concept, Use, Evaluate, and Ethics — so the two measurement approaches could be compared directly rather than studied in isolation. Confirmatory factor analysis established construct validity and reliability, and latent profile analysis grouped teachers by the pattern of agreement or disagreement between their self-reports and objective performance, including how prior AI literacy experience shapes that relationship.

    Implications for AI in Education

    The low correlation between self-reported and objective-based factors is a strong argument for performance-based assessment of AI literacy in teacher preparation and evaluation, where self-report surveys remain common. For Assessment Validity, the study shows that what educators say about their AI skills is not a reliable proxy for what they can do. The six-profile structure gives Faculty Development programs a diagnostic basis for targeting support — for example, distinguishing overestimating teachers who need calibration from low-SR/low-OB novices who need foundational experience — and it demonstrates how validated instruments can feed Learning Analytics pipelines that tailor interventions to teacher subgroups.

    Connected Concepts

  • Teacher AI Competency
  • Metacognition
  • K 12 AI Education
  • Affective Tutoring
  • Automated Essay Scoring
  • Plagiarism Detection
  • Help Seeking
  • Bias Mitigation

    • Connected Articles

  • Persistent AI Agents Academic Research — Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study
  • Beyond Detection Authentic Assessment AI 2025 — Beyond Detection: redesigning authentic assessment in an AI-mediated world
  • AI Changing Teaching Workflows — How AI Is Changing Teaching Workflows
  • Edumirror Educational Social Dynamics — EduMirror: Modeling Educational Social Dynamics with Value-driven Multi-agent Simulation
  • Authentic Products Authenticated Processes 2026 — From authentic products to authenticated processes: authentic assessment in AI-rich higher education
  • Hybrid E Assessment Semi Automated Grading — Hybrid E-Assessment in Higher Education: Semi-Automated Grading of Paper-Based Written Examinations

    • Citation

    Zhang, S., Xiao, R., Botelho, A. F., Liao, G., Chiu, T. K. F., Stamper, J., & Koedinger, K. R. (2026). How to Assess AI Literacy: Misalignment Between Self-Reported and Objective-Based Measures. arXiv:2601.06101.