Educational LLM Alignment

Hardy & Kim (2026) identify a cascading proxy problem in AI-for-education evaluation:

The gap between what LLMs are capable of and what actually benefits learners — benchmark performance, downstream task quality, and intended educational impact are three distinct and often-misaligned levels.

The Three-Layer Alignment Problem

Hardy & Kim (2026) identify a cascading proxy problem in AI-for-education evaluation:

1. Benchmark alignment (MMLU, pedagogical knowledge tests) — what models are typically evaluated on.

2. Downstream task alignment (expert human ratings of teaching quality) — what models are asked to do.

3. Intended impact alignment (student learning gains / VAMs) — what actually matters.

The paper demonstrates these three layers are not just loosely coupled — they can be negatively correlated.

Empirical Evidence

Study Design

Finding 1: LLMs Share a Homogeneous "Pedagogy Heuristic"

LLM-LLM agreement is substantially higher than LLM-human agreement. Models converge on a shared latent heuristic of "good teaching" that doesn't match expert human distinctions. This is attributed to shared pretraining on Internet text lacking authentic classroom discourse.

Finding 2: Benchmark Alignment ≠ Student Impact

Some models align moderately with expert ratings, but alignment with student learning gains is often near zero or negative. Human raters show a real (τ ≈ 0.11–0.14) signal with VAMs; LLMs largely don't. Reasoning-enhanced variants (o1, DeepSeek-R1) showed no improvement.

Finding 3: Ensembles Amplify Misalignment

Both benchmark-weighted aggregation and unanimous-voting ensembles worsened alignment with learning. Aggregating multiple misaligned models compounds the problem rather than averaging it out.

Finding 4: Model/Prompt Selection = 15% of Error

Choice of LLM and prompting strategy accounts for only ~15% of misalignment. The remaining ~85% is shared across all models — common pretraining data and objectives are the dominant driver. Prompt engineering and model selection are weak levers.

Broader Implications

1. Stop benchmarking alone — High scores on MMLU or even pedagogy-specific benchmarks do not predict beneficial educational impact. See Teachbench LLM Teaching Evaluation for syllabus-grounded alternatives.

2. Ensembles are not a safety net — When models share the same flawed pretraining priors, voting and weighting make things worse.

3. Pretraining is the intervention point — The field's focus on post-hoc alignment (RLHF, prompting) misses that shared pretraining corpora embed the core misalignment. See Pedagogical LLM Training for training approaches.

4. Measure impact directly — Practitioners must evaluate against intended student outcomes, not proxy task accuracy. Connects to Stanford Evidence Base AI K12 2026 demands for causal evidence.

This finding is a deep challenge to the ITS effectiveness literature: if even the best models can't align with student learning, what does "effective" tutoring AI look like? It also reinforces the Tutoring Specific Vs General AI finding that general-purpose AI underperforms pedagogically-designed systems.

Open Questions

Connected Concepts

Connected Articles

Citation

Hardy, M., & Kim, Y. (2026). Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact. arXiv:2603.00883.