Refining Collaborative AI Literacy: A Systematic Review of Human–AI Collaboration Competency and AI Literacy Measures

Synthesis: Yang, Kim, Jeong, and Lim (2026) ask whether existing measures of AI Literacy capture the ability to work with AI as a partner rather than as a tool. A PRISMA-guided search of five databases returned 5,355 records, and 17 studies of newly developed or adapted scales, covering 13,273 participants in higher education, met the inclusion criteria. Only three studies addressed human-AI collaboration explicitly, only one measured it as a construct alongside AI literacy, and 11 showed no representation at all; in two cases collaboration items were written and then removed during scale development. The authors read this as a boundary problem rather than a sampling accident: AI literacy instruments describe knowledge and tool use, while collaboration depends on coordination, communication, and Metacognition during interaction. They propose those three sub-factors as the basis for a distinct collaborative AI literacy, a conceptualization the review does not itself validate.

Key Findings

1. Seventeen studies, 13,273 participants. From 5,355 records across five databases, 17 studies of newly developed or adapted scales met the criteria, with sample sizes ranging from 158 to 2,713 and most samples drawn from universities.
2. Collaboration is almost never a construct. Three of the 17 studies operationalized human-AI collaboration explicitly, three embedded it implicitly in item wording, and the remaining 11 showed no representation at either the construct or the item level.
3. One instrument measured both. Only Wu et al.'s Interdisciplinary Competence in AI-Enabled Learning (ICAIL) measured AI literacy and collaboration competency together; its five Collaborative Problem Solving items name coordinating roles in human-AI workflows and negotiating task ownership.
4. Collaboration items get deleted. S4's collaboration and communication sub-constructs were removed after expert review, and S8's item "I can communicate effectively with AI tools based on the goals of my L2 writing" was dropped after factor analysis.
5. Self-report dominates the evidence base. Fourteen of 17 studies (82.4%) used self-report instruments and three (17.6%) used performance-based tests; 15 (88.2%) reported Cronbach's alpha, but only 9 (52.9%) reported convergent and discriminant validity.
6. The proposed construct is untested. The review offers coordination, communication, and Metacognition as the core of collaborative AI literacy, but states that this conceptualization has not yet been validated through scale development.

How the review was built

The design follows PRISMA 2020. Five databases (Web of Science, Scopus, ERIC, PsycINFO, and IEEE Xplore) were searched in January 2026 for English, peer-reviewed studies published since 2020 that developed or substantially adapted a scale measuring AI literacy or human-AI collaboration competency among higher education students. Translation-only validations, K-12 and professional samples, teacher-focused studies, and intervention-effectiveness studies were excluded. Screening ran from 5,355 records to 1,753 duplicates removed, 2,387 title exclusions, 1,178 abstract exclusions, and 37 full texts, of which 20 were excluded, leaving 17 studies; inter-rater agreement at screening was a Fleiss' kappa of .78. Data were synthesized narratively, with each scale coded at the construct level and then at the item level into one of three categories: explicitly operationalized, implicitly embedded, or not represented. Methodological quality was judged from reported psychometric evidence rather than a full COSMIN appraisal, a decision the authors justify by the early stage of measurement development in the field.

Collaboration at the edge of AI literacy scales

The 17 instruments are a recognizable sample of the AI literacy literature: 11 measure AI literacy alone, one (S11) measures AI literacy and collaboration together, two measure AI empowerment, two AI acceptance, and one generative AI reliance. The self-report format dominates. Nine studies built their scale by integrating more than three earlier instruments and six developed items anew, so the integrated scales carry earlier frameworks' blind spots forward. Where collaboration appears, it appears at the margins. S11 is the exception: its Collaborative Problem Solving sub-construct treats AI as a teammate, with items on coordinating roles, mediating conflicts, and evaluating AI contributions to team outcomes. S4 grouped collaboration, communication, social interaction, and anthropomorphic interaction across its affective and behavioral dimensions, but only GAI anthropomorphic interaction survived expert review. S8 validated a communication item under a "use" sub-construct and then deleted it. Placement is inconsistent across scales as well, with comparable items filed under "use" in one instrument and "autonomous learning" in another. Several studies that mention collaboration turn out to measure something else: using AI to improve communication among human teammates (S5, S9, S10, S15) is human-human collaboration facilitated by a tool, not collaboration with the tool.

Tool use, joint work, and where the boundary falls

Definitions are not the bottleneck. Nine of the 17 studies cite Long and Magerko's definition, which explicitly includes the capacity to "communicate and collaborate effectively with AI," yet the four sub-constructs most scales borrow from Ng et al. contain no such dimension. The review's distinction is between AI as an instrument that supports a task and AI as a participant in interdependent problem solving, where the learner sets goals, supplies context, interprets output, and monitors the exchange. That framing draws on distributed cognition and hybrid intelligence: generative AI produces and transforms material while the human handles judgment and integration, so quality depends on the interaction rather than on technical proficiency alone. The cognitive, affective, and behavioral dimensions used across AI literacy frameworks do not settle the question either, because collaboration items land in different dimensions depending on the scale, and the cognitive dimension is the least articulated, rarely reaching joint processes such as building shared understanding. The authors also caution against anthropomorphism as a shortcut, citing students in one focus group who recognized that communicating with a machine differs from communicating with a person.

Three sub-factors and the wording problem

The review's constructive contribution is a proposal to measure collaborative AI literacy through coordination, communication, and Metacognition. Coordination covers how roles and interaction flow are managed during joint work, and the authors argue that "interaction" is too broad a term to carry the construct unless a scale specifies which form it means, such as dialogue, transmission, tool use, or experience. Communication covers supplying context and interpreting responses, and the turn toward conversational systems makes reciprocal dialogue a plausible object of measurement rather than prompt craft alone. Metacognition covers monitoring and adjusting one's own thinking, including the Learner Agency to evaluate AI output before integrating it, and connects to concerns about automated reliance undermining critical evaluation. Wording matters at the item level: verbs such as "use," "useful," "aid," and "assist" reinforce an instrumental reading, while "with" positions AI as a partner. Their illustrative revision turns "identify errors made by AI" into a reciprocal item that also asks whether AI helps identify errors in the learner's work. Problem-solving and generative AI contexts, they suggest, are more likely than general-use contexts to elicit the coordination the construct requires.

What this means for practice

• Instructors. Teach coordination and communication as course content, not as a byproduct of tool access: have students assign roles in an AI-supported task, state what context the model needs, and evaluate what it returns.
• Assessment designers. Do not read an AI literacy score as evidence of collaborative capability. If collaboration is the outcome, assess it in a Problem Solving task where the interaction can be observed rather than inferred from a self-report item.
• Instructional designers. Build tasks with a visible division of labor and a communication artifact, such as a revised prompt and a short account of why the output changed, so that coordination and metacognition leave a trace.
• Researchers. Treat the three sub-factors as hypotheses. A new instrument needs construct definitions, items that signal two-way interaction, and convergent and discriminant evidence, not alpha alone, and a framework should keep collaboration named rather than folding it into technical knowledge.

Limitations

• The review includes 17 studies, all English-language and peer-reviewed, with no gray literature and no citation or hand searching, so relevant instruments may be missing.
• Widely cited instruments validated only in K-12 or professional settings were excluded by design, which narrows the inventory of AI literacy measures considered.
• Quality was appraised from reported psychometrics rather than a full COSMIN assessment, so weak reporting and weak measurement cannot be fully separated.
• The 13,273 participants are responses rather than unique people: two studies (S9, S14) sampled overlapping groups at two time points.
• The collaborative AI literacy conceptualization is theoretical and has not been tested through scale development or construct validation.

Citation

Yang, J. W., Kim, J., Jeong, S., & Lim, J. (2026). Refining Collaborative AI Literacy: A Systematic Review of Human–AI Collaboration Competency and AI Literacy Measures. Online Learning, 30(SI2), e5816.