Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction

Synthesis: Nanayakkara and Halloluwa (2026) address the fundamental challenge of objective learning assessment by benchmarking fifteen machine learning and deep learning models for EEG-based familiarity prediction across two cognitive domains — human faces (factual knowledge) and mathematical equations (conceptual knowledge). Using continuous EEG from 23 participants and spectral features across six frequency bands, they show that standard stratified cross-validation yields artificially high performance (up to 0.9853 F1 with a CNN) due to temporal leakage, whereas rigorous trial-independent Group K-Fold validation drops the peak to 0.6038 F1 — still statistically significant above chance. The study establishes a realistic benchmark for EEG-based cognitive monitoring in educational technology and cautions against overestimating model generalizability in Automated Assessment.

The promise and pitfalls of neurophysiological assessment

Traditional assessment methods such as quizzes and exams are indirect proxies for learning, relying on a learner's ability to articulate knowledge. The paper motivates psychophysiological signals — eye-tracking, skin conductance, and especially electroencephalography (EEG) — as direct, non-invasive windows into the neural correlates of knowledge acquisition. EEG offers high temporal resolution, making it promising for detecting familiarity, a foundational element of learning. The study positions itself as a comprehensive Benchmark filling a gap: prior work often focused on a limited set of models or a single cognitive domain, so this work evaluates fifteen models across two distinct domains.

Methodological rigor: trial-independent evaluation

The central methodological contribution is the contrast between two validation schemes. Standard stratified cross-validation allows temporal leakage across neighboring epochs, producing inflated estimates (up to 0.9853 F1 with CNN). A rigorous trial-independent validation (Group K-Fold), which respects the temporal structure of the data, drops peak performance to 0.6038 F1 (CNN) — still statistically significant above the 25% chance level. This demonstrates the critical necessity of trial-independent evaluation to avoid overestimating model generalizability, a lesson directly relevant to AI evaluation and the limitations of AI in education research.

Neural biomarkers and feature importance

Beyond classification performance, the authors use feature importance and SHAP analysis to identify temporal and frontal Gamma and Beta oscillations as the most critical biomarkers for familiarity. This connects the benchmarking to the underlying neural signatures of learning and suggests which brain signals carry the most information about whether a learner recognizes familiar content.

What this means for practice

• Designers. Validate EEG-based assessment models with trial-independent Group K-Fold rather than standard stratified cross-validation: stratified CV inflated a CNN to 0.9853 F1, while trial-independent validation dropped the peak to 0.6038 F1, still above the 25% chance level.
• Designers. Prefer ensemble methods over deep learning when training on small EEG datasets, since Gradient Boosting and Random Forest generalized better to unseen trials than the deep architectures.
• Designers. Report fold-level spread rather than the peak alone: the domain-separated LOGO results carried very large standard deviations (0.8670 +/- 0.2392 for equations and 0.8875 +/- 0.2347 for faces under CNN).
• Designers. Start from temporal and frontal Gamma and Beta oscillations, which feature importance and SHAP analysis flagged as the most informative biomarkers, when building lighter real-time familiarity detection.

Limitations

• Sample size and homogeneity: 23 participants, all with a STEM background, limit generalizability, and the strongest faces-only claims rest on a small 13-block subset.
• Cross-subject leakage remains possible because participant identifiers were not retained, so Group K-Fold could only block at the trial level and one subject's trials may appear in both training and test sets.
• LOGO results are unstable: each held-out fold is a single trial block, standard deviations run near 0.24, and the higher CNN point estimates were not independently permutation-tested.
• Spectral resolution is coarse (128 Hz sampling with 32-sample Welch segments gives 4 Hz bins), making narrow bands such as Delta (1-4 Hz) hard to isolate, and manual ICA selection and artifact rejection add subjectivity.

Citation

Nanayakkara, I., & Halloluwa, T. (2026). Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction.