OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models

Synthesis: Synthesis: Lee and thirteen co-authors present OpenLearnLM Benchmark, an open framework that scores educational large language models on three axes drawn from educational assessment theory: Knowledge, Skills, and Attitude. Knowledge covers curriculum-aligned content and pedagogical understanding through 2,304 multiple-choice items, mostly reused from established sources rather than generated. Skills measures situated practice through 122,425 items arranged in a four-level hierarchy of Centers, Roles, Scenarios, and Sub-scenarios, with difficulty tiers set by Bloom's cognitive taxonomy and Krathwohl's affective taxonomy. Attitude examines epistemic, instructional, and normative-ethical stances and adds a deception test adapted from Alignment Faking methodology, comparing behavior under monitored and unmonitored prompts. Evaluating seven frontier models, the authors find no model dominant across axes: Claude-Opus-4.5 leads practical skills while scoring lowest on content knowledge, Grok-4.1-fast leads content knowledge but shows the weakest alignment consistency, and four models keep consistent behavior regardless of perceived oversight. Cross-axis correlations are weak or negative, so single-axis evaluation cannot establish readiness for educational settings, and the benchmark is released openly for the research community.

Key Findings

• Capability is multidimensional. Claude-Opus-4.5 records the highest Skills average (8.82) despite the lowest content knowledge (66.3%), while Grok-4.1-fast leads content knowledge (86.5%) and posts the worst deception score (5.50).
• Content and pedagogical knowledge diverge. Claude-Opus-4.5 shows a 19.8 point gap between content (66.3%) and pedagogical knowledge (86.1%), suggesting it understands how to teach better than what to teach.
• Skills cluster by educational center. Counseling averages highest (8.78), with GPT-5.2 reaching 9.07 there, while Assessment and Research are hardest at 8.42 each, pointing to structured judgment as the weak spot.
• Alignment consistency separates models sharply. GLM-4.7, GPT-5.2, DeepSeek-v3.2, and Claude-Opus-4.5 score 1.0, while Grok-4.1-fast (5.50) and Gemini-3-Pro (5.00) behave more permissively when they consider themselves unobserved.
• The three axes are largely independent. Knowledge correlates negatively with Skills (r = -0.51) and Attitude (r = -0.63), while Skills and Attitude correlate weakly (r = 0.22), so accuracy alone says little about educational capability.

The gap in prior educational benchmarks

Earlier educational benchmarks grew quickly but stayed narrow. LearnLM grounds evaluation in learning science principles yet releases no public dataset; EduBench covers nine scenarios across 18.8K items without hierarchy; MathTutorBench and TutorBench concentrate on tutoring interactions. The authors trace three recurring limits: no grounding in educational theory, no hierarchical task structure, and a narrow focus on knowledge. They also cite a gap between accuracy and teaching quality, with one tutoring system reaching 97.3% answer accuracy while only 56.6% of its responses were pedagogically sound. No prior benchmark defined educational roles or checked whether behavior changes under perceived monitoring. OpenLearnLM answers those gaps with a shared structure: three axes, a four-level taxonomy of Centers, Roles, Scenarios, and Sub-scenarios, and an attitude test borrowed from alignment research.

How the benchmark was built and scored

Construction rested on 14 experts and a four-month Delphi study that fixed the taxonomy first. Skills items came from an automated gpt-5-mini pipeline producing 122,425 items at a 99.6% success rate, split into roughly 116,000 training and 6,100 test items. Knowledge items stayed authentic: 798 from C-Eval, 120 from GPQA, 243 from a Korean teacher examination, and 1,143 from the Chilean pedagogy benchmark, totaling 2,304 multiple-choice questions. An LLM-as-Judge screened quality against a 25-point rubric, and 122,365 of 122,439 items passed. Knowledge is graded by exact match; Skills long answers against a 10-point rubric for each of the 81 sub-scenarios; the 12 standard attitude scenarios with 10 samples each; and the 2 deception scenarios with 100 samples under monitored and unmonitored prompts, where the deception score is the difference between the two means.

Model profiles and why the axes stay separate

Across the seven frontier models the axes rarely move together. Grok-4.1-fast posts the strongest content knowledge (86.5%) while Gemini-3-Pro leads pedagogical knowledge (89.6%). Claude-Opus-4.5 has the widest content-to-pedagogy gap at 19.8 points yet the best Skills average (8.82), and GPT-5.2 reaches the highest single center score in Counseling (9.07). GLM-4.7 and GPT-5.2 hold the most balanced profiles, staying above average on all five plotted dimensions, and no model leads everywhere. Correlations across the ten dimensions are r = -0.51 between Knowledge and Skills, r = -0.63 between Knowledge and Attitude, and 0.22 between Skills and Attitude, which the authors read as a possible trade-off. A knowledge score alone cannot establish whether a model is ready for a given educational role.

What this means for practice

• Evaluate per use case, not by one leaderboard. The strongest content model may be the weakest on practical tasks; match models to the work they will carry.
• Treat deception scores as a deployment gate. A model that behaves differently when unmonitored needs extra Guardrails or logging before unsupervised student contact.
• Watch the content-to-pedagogy gap. A model strong on teaching theory but weak on subject facts needs content validation, not trust in its explanations.
• Expect structured judgment to be hardest. Rubric application, error diagnosis, and research reasoning score lowest, so keep humans in the loop.

Limitations

• The attitude axis rests on a small item pool. Fourteen scenarios, and only 2 for deception, carry the behavioral argument, so per-model alignment conclusions stay thin.
• Most Skills items are machine-generated. The 122K items come from a gpt-5-mini pipeline filtered by an LLM judge, so quality rests on automated screening alone.
• Items are borrowed and translated. Content items come from Chinese examination and graduate science benchmarks, and pedagogical items from Chilean and Korean examinations, which may carry cultural assumptions.

Connected Concepts

• Benchmark
• AI Ed Evaluation
• Large Language Models (LLMs)
• Assessment Validity
• Educational Measurement
• Trust Calibration

Connected Articles

• Benchmarking the Pedagogical Knowledge of Large Language Models
• The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness
• Toward Convergence in Student-LLM Interactions: A Rapid Scoping Review and Taxonomy for Learning-Oriented Use
• Distinguishing performance gains from learning when using generative AI

Citation

Unggi Lee, Sookbun Lee, Heungsoo Choi, Jinseo Lee, Haeun Park, Younghoon Jeon, Sungmin Cho, Minju Kang, Junbo Koh, Jiyeong Bae, Minwoo Nam, Juyeon Eun, Yeonji Jung, and Yeil Jeong (2026). OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models. arXiv:2601.13882. https://doi.org/10.48550/arXiv.2601.13882