Tracing GenAI Literacy: Student-AI Interaction Patterns in Academic Writing

Synthesis: Identifies interaction signatures of Large Language Models (LLMs) literacy using Epistemic Network Analysis (ENA) on logs from 162 students. High-literacy students exhibit iterative, strategic refinement and dense cognitive networking, while low-literacy students rely on direct, linear commands. This work emphasizes that AI Literacy is a developmental capacity requiring structured Scaffolding and Prompt Engineering discipline. It connects to the need for Curriculum Design that targets Metacognition and Agentic AI rather than just syntax mastery.

Key Findings

• Interaction logs from 162 university students engaged in a GenAI-assisted abstract writing task were analyzed using Epistemic Network Analysis (ENA).
• High-literacy students exhibit iterative refinement and strategic questioning, producing distinct interaction signatures in the process data.
• Low-literacy students rely on direct generation commands, characterized by transactional, generation-oriented dependence on the tool.
• GenAI literacy is not just a static score but a dynamic behavioral capability that shapes the human-AI collaboration process.
• The work paves the way for data-driven literacy assessment and real-time interventions based on observed interaction behavior rather than self-report.

Study Design & Method

A total of 162 university students (M_age = 20.1) participated in a GenAI-assisted abstract writing task. Prior to the task, students completed a validated GenAI Literacy Test assessing technical understanding, interaction skills, and related constructs. Using Epistemic Network Analysis, the researchers modeled and compared the questioning strategies of students with varying GenAI literacy levels, examining how literacy manifests in the structure of actual human-AI collaboration. The study was presented at the First International Workshop on Advancing AI Literacy with Learning Analytics (AI-LIT) at LAK26.

What this means for practice

• Instructors. Assess AI Literacy from interaction process, not a single questionnaire score: logs from 162 students, split at the median of a validated GenAI Literacy Test into a high group (n = 89) and a low group (n = 73), produced structurally distinct questioning networks under Epistemic Network Analysis.
• Instructors. Ask students to draft their own summary first and request critique of it, then to refine iteratively: high-literacy networks linked AI Improvement Commands with Clarification Questions and Meta-Commands, an iterative sense-making-then-refinement cycle, whereas low-literacy students paired broad generation commands with basic fact retrieval.
• Designers. Use interaction constraints that force synthesis rather than adoption: the study's custom platform imposed a 30-word limit per prompt and disabled copy-pasting, a directly reproducible lever for other tools.
• Instructors. Treat low-literacy challenge questions as a teaching opening: the challenging-question link appeared in the low-literacy network mainly as a reaction to confusion or hallucinations rather than strategic critique, so make explicit how to question a model's logic.
• Designers. Build Learning Analytics dashboards around the observed behavioral markers — the shift from direct generation commands toward iterative refinement — to trigger real-time support instead of end-of-course self-report.

Limitations

• Literacy grouping came from a median split on students' self-reported GenAI Literacy Test results rather than from an externally validated performance measure, yielding a high group (n = 89) and a low group (n = 73) within a single cohort.
• Participants were 162 university students (M_age = 20.1) from one institution working on one academic abstract-writing task in one custom platform built on DeepSeek, so the interaction signatures may not transfer to other tasks, disciplines, or tools.
• The paper reports preliminary findings presented at the AI-LIT workshop at LAK26, and its coding framework was adapted from prior work; human coding of a subset reached substantial but not perfect agreement (Cohen's κ = 0.75).
• Because the analysis infers intent from behavioral codes alone, a challenge question logged as confusion-driven criticism and one logged as strategic critique are not distinguishable in the logs.

Citation

Angxuan Chen & Jiyou Jia (2026). Tracing GenAI Literacy: Student-AI Interaction Patterns in Academic Writing.