Gaze-Informed Proactive AI Assistance for Children’s Picture Exploration

Synthesis: Ollie, a gaze-informed AI assistant for children's picture exploration, uses eye-tracking to trigger Large Language Models (LLMs) narrative descriptions; within-subject experiment shows gaze-informed assistance keeps children's attention longer and guides exploration effectively.

Key Contributions

• Ollie, a gaze-informed AI assistant for children's picture exploration, uses eye-tracking to trigger LLM narrative descriptions; within-subject experiment shows gaze-informed assistance keeps children's attention longer and guides exploration effectively.

Connections to AI in Education

This paper contributes to the growing body of research on AI applications in educational settings, specifically in the domains of AI in Education, Intelligent Tutoring, and Equity. The findings have implications for how educators design learning experiences that leverage AI while maintaining appropriate pedagogical oversight.

What this means for practice

• Designers. Ground proactive assistance in the learner's own attention rather than a fixed clock: gaze-triggered narration held children on the narrated region significantly longer (6.48 s vs. 1.67 s in the baseline) and drew them to the adjacent region more often (54.06% vs. 35.48%), with the secondary region chosen closer to current focus (relevance distance 0.43 vs. 0.52).
• Designers. Budget interventions and keep them spaced: in the pilot, children moved on after about three minutes and engagement declined after roughly three assistance instances, so the main study gave three narrated regions per image with a minimum three-second interval between triggers. Treat eye-tracking data as sensitive by default, since the one parent who preferred the non-gaze version did so over privacy and security concerns about eye tracking.
• Instructors. Use proactive narration as situational support rather than routine practice, as the participating kindergarten teacher recommended for rest periods, specific projects and older preschool children, and add question answering before relying on it: parents and the teacher identified the inability to answer children's questions as the main limitation.
• Researchers. Do not read stated preference as evidence of learning: 15 of 22 children preferred the gaze condition but 11 of the 22 could not explain why, and only 8 of the 22 usually read with a parent, so preference and comparison data need behavioral and comprehension measures behind them.

Limitations

• The final sample was 22 children (11 five-year-olds, 4 six-year-olds, 4 seven-year-olds, 3 eight-year-olds), with 21 contributing valid gaze data; 4 of the 26 recruited children were lost (two declined to finish, one session failed technically, one child left early).
• Areas of interest were defined manually in advance, which the authors state limits scalability and depends on annotators' judgments about what counts as a meaningful region.
• Assistance is one-way narration: the system cannot answer questions or explain unfamiliar words, which reduces its usefulness in realistic reading situations, and the authors additionally report narration quality problems in grammar, pronunciation, repetition, delay and voice naturalness, especially for languages other than English (the study was run in German).
• One participant was excluded and 13.2% of gaze data removed by the tracking-quality filter, so the attention analysis rests on a filtered subset of the collected sessions.

Citation

Zekun Wu, Man Su, Huiyong Li, Tomohiro Nagashima, Anna Maria Feit (2026). Gaze-Informed Proactive AI Assistance for Children’s Picture Exploration. submitted 1 Jul 2026