Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

Synthesis: The ALICE benchmark from Sun and colleagues at the DIPF and IPN Leibniz institutes reframes automatic short answer scoring as a Benchmark built from real science education classroom answers. Its 16,572 German student responses to 112 questions carry teacher-designed rubrics at three levels: overall learning performance, mastery of individual knowledge elements, and the presence of epistemic skills such as reasoning or claim formulation. The authors contrast three ways to score an answer — sequence classification, zero-shot LLM prompting, and rubric-retrieval, in which the model picks the best-matching rubric item for each response. Educational NLP encoders and Machine Learning classifiers do the scoring; rubric-retrieval generalises best to unseen questions, and lightweight decoder-based encoders edge out masked language models. ALICE is genuinely hard: even the strongest systems trail human agreement on the fine-grained knowledge-element and skill subtasks, and one-shot prompting closes the gap only on the easiest subtask.

Key Findings

1. A large, rubric-annotated German corpus. ALICE holds 16,572 student answers to 112 scientific-education questions, expanded from the earlier ALICE-LP 1.1 release of about 13K answers across 117 questions, with new knowledge-element and skill annotations.
2. Three complementary scoring subtasks. ALICE-LP rates how well an answer addresses the question (incorrect, partially correct, correct); ALICE-KE rates use of target concepts; ALICE-SK rates epistemic behaviors such as reasoning or claim formulation.
3. Rubric levels overlap, so labels are not mutually exclusive. Entailment checks show adjacent levels usually overlap: 91% of LP level-1-to-2 pairs and 95% of skill level-1-to-2 pairs entail, so a strong answer may also satisfy parts of a lower rubric.
4. Rubric-retrieval wins on unseen questions. Framing scoring as selecting among question-specific rubric candidates, rubric-retrieval outperforms fixed-label sequence classification consistently on Test-UQ, with the largest margins on ALICE-KE and ALICE-SK.
5. Lightweight LLM encoders lead. Under rubric-retrieval, Llama-3.2-3B-Instruct reaches 69.5 macro-F1 on ALICE-LP with question and solution context; Llama-3.2-3B is best on ALICE-KE and ALICE-SK, beating masked language model baselines on unseen questions.
6. Zero-shot LLMs trail supervised models. GPT-5-mini prompting approaches fine-tuned performance only on ALICE-LP (66.1) and ALICE-KE (59.9); smaller open-weight models fall further behind, and adding full question context can even hurt them.

A benchmark built from how teachers grade

ALICE is a German-language Benchmark for automated assessment of short constructed responses in science education. Its design starts from a gap the authors identify in educational NLP: most public datasets score whether a student answered the question, not whether the answer shows mastery of the underlying concepts or the epistemic behaviors teachers care about. Each ALICE instance bundles the question prompt, a sample solution, the student answer, and per-level rubrics. Where earlier work scores similarity to a reference solution, ALICE provides a qualitative rubric for every performance level, so partially correct and incorrect answers carry diagnostic signal. Annotations were designed by subject educators and produced by expert annotators on the INCEpTION platform, with validity checked through quadratic weighted kappa agreement reported per subject and dimension: 0.84 for biology knowledge elements, 0.86 for chemistry knowledge elements, and 0.72 for physics knowledge elements.

Why rubric-retrieval suits question-specific rubrics

The authors' central modeling proposal is to reframe scoring as rubric-retrieval: expand each answer into answer–rubric pairs and let a cross-encoder pick the best-matching rubric item, with a softmax computed only over the rubric candidates for that question. This differs from automated essay scoring pipelines that assume one fixed label space across all questions. Because questions have their own performance levels, and because the knowledge-element and skill rubrics are not additive, a fixed classifier must invent or omit levels. The cost of retrieval is scalability: each answer becomes several pairs, so training and inference are heavier than single-pass classification. The authors also show that rubric levels overlap — 87% of adjacent knowledge-element pairs between levels 2 and 3 entail — which is why binary matching is insufficient and the model must contrast the correct rubric against alternatives.

What the model comparison shows

Using lightweight LLM encoders rather than generative decoding, the benchmark separates three approaches on two test splits: unseen answers (Test-UA) and unseen questions (Test-UQ). Rubric-retrieval is competitive with sequence classification on ALICE-LP but pulls clearly ahead on ALICE-KE and ALICE-SK, especially when questions are unseen. Under rubric-retrieval, Llama-3.2-3B-Instruct reaches 69.5 macro-F1 on ALICE-LP with question and solution context, and Llama-3.2-3B leads ALICE-KE and ALICE-SK. Masked encoders such as mmBERT stay competitive in some fixed-label settings but do not overturn the trend that decoder-based encoders generalize better. Zero-shot prompting with stronger proprietary models approaches fine-tuned performance on ALICE-LP (GPT-5-mini at 66.1) and ALICE-KE (59.9) but lags on ALICE-SK.

What rubric text and context actually contribute

Ablations on input format isolate what helps. Rubric text is the strongest single signal: moving zero-shot prompts from bare label names to the full rubric description yields large gains for stronger models. Adding question and sample-solution context on top helps fine-tuned models most on ALICE-KE and ALICE-SK, whereas on ALICE-LP the extra context is largely redundant, since overall performance can be approximated from the sample solution. For smaller open-weight models, more context can degrade performance, which the authors read as an inability to use richer input selectively. Masked language models are also more prone than LLM encoders to degradation from extra context. This matters for formative assessment: the rubric wording itself, not model scale, carries most of the achievable gain.

What this means for practice

• Instructors. Give the model the rubric, not just the question: rubric wording is the strongest driver of score quality, so the level descriptors you write are doing the real work when Feedback or scoring is automated.
• Assessment designers. Design question-specific rubrics that describe observable levels; the benchmark shows that fixed, uniform label sets across questions lose accuracy exactly where diagnostic detail matters, on concepts and skills.
• Researchers. Treat ALICE as a public Benchmark for educational measurement and report per-subject and per-level results, because aggregate scores hide the intermediate levels where models struggle most.
• Developers. Prefer an encoder-based rubric-retrieval design over free-form generation for scoring: it is cheaper, more stable on unseen questions, and does not need token-by-token decoding.

Limitations

• German only. ALICE covers German-language scientific education, so results may not transfer directly to other languages or subject domains.
• Rubrics are a prerequisite. Rubric-based scoring relies on high-quality rubrics that may not be available in real-world settings, and the study does not test LLM-augmented rubric generation.
• Narrow benchmark scope. The study covers discriminative scoring only, not generative scoring or feedback generation, and mathematics questions carry no knowledge-element or skill annotations.
• Intermediate levels remain hard. Fine-grained knowledge-element and skill distinctions, especially intermediate levels, stay difficult for every evaluated model.
• Zero-shot LLMs are not a shortcut. Smaller open-weight models trail supervised approaches, and adding full context can degrade their performance.
• Retrieval costs scale. Expanding each answer into multiple answer–rubric pairs raises training and inference cost and limits scalability to larger models.

Citation

Sun, Z., Gombert, S., Lossjew, J., Wyrwich, T., Czinczel, B. K., Bednorz, D., Kubsch, M., Neumann, K., & Drachsler, H. (2026). Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring. arXiv preprint arXiv:2610.09661.