🧠 AI Ed Wiki

📄 Research Article

TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

◐ medium · 2026-06-03benchmark, cs-education, k-12, multimodal

Synthesis: Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors

Abstract

Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focuses on visual programming for productivity; it remains unclear how well current VLMs perform on education-oriented visual programming and what factors limit their performance. To bridge this gap, we introduce T URTLE AI, a benchmark containing 823 tasks curated based on real-world visual programming tasks in the Turtle Graphics domain. Solving these tasks requires models to perceive geometric patterns, reason about spatial relationships, and synthesize Python code that faithfully reproduces geometric patterns. We evaluate 20+ VLMs, including GPT-5, GPT-4o, and Qwen2-VL-72B, and find that they struggle significantly, with most achieving succes

Connected Concepts

  • Human In The Loop AI
  • Pedagogical LLM Training
  • Computational Thinking
  • STEM Education
  • Math Education
  • Educational Policy AI
  • Formative Assessment
  • Affective Tutoring

    • Connected Articles

  • Cstutorbench Slm Tutors — CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming
  • Privacy Aware Classroom Incident Recognition 2026 — Robust and Efficient Motion Reasoning for Privacy-Aware Classroom Incident Recognition
  • LLM Cognitive Diagnosis Handwritten Math — Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work
  • Vocabulary Difficulty Prediction — What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction
  • Anvil AI Educational Animations — ANVIL: Analogies and Videos for Lecturers
  • Eduzone LLM Safety K12 — EduZone: A Framework for Evaluating LLM Safety for K-12 Students and Teachers

    • Citation

    for, T.U.A.B.M.M., Graphics, V.P.I.T., Wen, C., Staub, J., tasks, T.U.A.A.B.C., visual, Y.E.A.L.E.A.B., & Rodriguez, Y.E.A.W.E.A. (2026). TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics. T URTLE AI, each task requires a VLM to generate Python code that reproduces a target image