A Comparative Study of Knowledge Component Granularity in Intelligent Language Tutoring Systems

Synthesis: El Hefny and Meurers report a large-scale comparison of 19 knowledge component (KC) models, spanning 11 levels of the FeedBook domain hierarchy, built on 153,819 first-attempt responses from 1,059 seventh-grade students learning English in German schools. Using the Additive Factors Model (AFM) and Performance Factor Analysis (PFA), two interpretable knowledge tracing models, they show that the best granularity depends on the modeling goal: fine-grained item-level models predict best for new students on known content, while semantically grounded mid-level models generalize best to new content. Model-fit criteria split along the same lines, with AFM's Bayesian Information Criterion (BIC) favoring mid-level semantic models because of how its fixed-effects parameterization penalizes the many parameters of fine-grained student models.

Key Findings

1. Fine-grained models win when generalizing to new students on known content. In student-blocked cross-validation, Actionable Element ID achieved the best AFM performance (RMSE = 0.4083, AUC = 0.7617), almost tied with Item ID (RMSE = 0.4085), with both implementations agreeing on the ranking.
2. Semantic mid-level models win for new content. In item-blocked cross-validation, Property ID + Exercise Type (Multi-skill) ranked first under AFM (RMSE = 0.4243), while fine-grained identifiers collapsed: Actionable Element ID fell from 1st to 17th (RMSE = 0.4447) because its KC values never recur across held-out elements.
3. Model fit and complexity criteria disagree. Under AIC both AFM and PFA favored fine-grained models — Exercise ID (AIC = 151004.18) under AFM and Actionable Element ID (AIC = 148742.47) under PFA — but under AFM BIC the best model was Property ID + Exercise Type (Single-skill), BIC = 172952.26, while Actionable Element ID was penalized to BIC = 233856.01 because student and KC effects are estimated as fixed effects.
4. Learning dynamics are granularity-dependent. Fine-grained models showed near-zero mean learning rates (Actionable Element ID µγ = 0.000; Item ID µγ = −0.001) partly from sparse repeated opportunities, while Exercise ID (µγ = 0.126) and Property ID + Exercise Type (Single-skill) (µγ = 0.058) showed positive practice signals.
5. Filtering for annotation coverage was substantial but concentrated. Strict property mapping initially removed 45.7% of records; a newer annotation tool variant reduced this to 30.6%, with the combined label positive|statement alone accounting for 74.9% of excluded records.

Representation choice is a modeling decision

The paper treats KC granularity as an empirical question rather than a fixed property of the domain. FeedBook, a web-based intelligent tutoring system for teaching English as a foreign language, implements a five-level domain hierarchy — learning units, learning targets, realizations, properties, and linguistic constructs — developed collaboratively by linguistic and pedagogical experts within the AI2Teach project. From it, the authors derived 19 candidate models across 11 categories, from fine-grained identifiers (Actionable Element ID, Item ID, Item Set ID, Exercise ID) through semantically grounded mid-level representations (Linguistic Constructs, Properties, Property ID + Exercise Type) to coarse curriculum-level groupings (Realization, Learning Target, Learning Unit) and a single-KC baseline.

The data come from the fourth of four randomized field trials around the system, AI2Teach, run from February 2024 to mid-July 2024 with 1,199 students in 52 school classes, reduced after filtering to 153,819 first-attempt records from 1,059 students with 71.8% correct on the first attempt. Actionable elements are automatically annotated with linguistic constructs from the English Grammar Profile via the rule-based POLKE tool, and the study used the AFM Optimized Python implementation from LearnSphere, validated against the DataShop benchmark Geometry Area (1996-97) dataset across 38 KC models with a mean absolute difference of 0.29.

The transfer trade-off

The core result is a tension between fitting known content and transferring to unseen content. When new students answer exercises already in the system, the finest granularity wins: item-level identifiers absorb item-specific difficulty and near-memorization effects, and both AFM and PFA place Actionable Element ID and Item ID at the top under all three metrics. When the task is predicting performance on held-out actionable elements, the same models fall to the bottom of the ranking — for example, Item Set ID dropped to 19th under AFM (RMSE = 0.4566) — because an identifier that appears only inside a single held-out element contributes nothing from the training folds.

Semantic mid-level models occupy the space between these extremes. Property ID + Exercise Type combines a linguistic property with the exercise format used to practice it, and it led the item-blocked ranking under AFM (Multi-skill RMSE = 0.4243, Single-skill RMSE = 0.4252). Under PFA, the fine-grained Exercise ID model ranked first (RMSE = 0.4161) while the same semantic variants took the next two positions, showing that a fine-grained model can survive when its KC values recur across content. Coarse curriculum-level models never led: Learning Target and Learning Unit models ranked near the bottom in both cross-validation settings, too broad to capture response variation.

Why the criteria disagree

The information criteria illustrate how implementation choices shape the apparent answer. AIC, with its moderate complexity penalty, favored fine-grained models under both AFM and PFA, though the two implementations did not select the same model. BIC penalized complexity much harder, and because the Python AFM implementation models student and KC effects as fixed-effect columns, its parameter count grows with the number of students and KCs — producing the huge BIC gap that pushed fine-grained models to the bottom (Actionable Element ID at 233856.01 versus 172952.26 for the mid-level winner). The R-based PFA implementation instead uses random effects and a smaller parameter set, so its BIC continued to favor fine-grained models. The authors therefore interpret the criteria only within each implementation, and treat the divergence itself as a finding about how complexity is represented.

Learning-rate and learning-curve analyses give a related diagnostic view. Near-zero rates for fine-grained models partly reflect the opportunity distribution their granularity creates: dividing the same interaction data among more KCs leaves few repeated observations per component. Among the eight semantic mid-level models, Property ID + Exercise Type (Single-skill) showed the clearest declining predicted-error trajectory, consistent with its largest positive mean learning rate among semantic models; the authors caution that these analyses describe alignment with observable practice patterns rather than verified skill acquisition.

What this means for practice

• Choose granularity by the modeling task, not by tradition. If the goal is predicting familiar items for new students, fine-grained item-level KCs are appropriate; if the system must sequence activities the learner has not seen, semantic mid-level KCs such as Property + Exercise Type generalize better (AFM item-blocked RMSE 0.4243 vs. 0.4447 for Actionable Element ID).
• Evaluate candidate KC models the way this study did, with student-blocked and item-blocked cross-validation side by side. A model that wins only on student-blocked prediction may be memorizing item identifiers rather than tracking transferable knowledge, which matters for adaptive exercise selection.
• Expect fit criteria and cross-validation to disagree. AIC, BIC, and learning rates answer different questions, and the BIC ranking depends on whether student and KC effects are fixed- or random-effects; report and interpret each within its own framework.
• Audit annotation-driven filtering before trusting semantic KCs. The 45.7% initial record reduction and its concentration in a few frequent combined labels (positive|statement at 74.9% of exclusions) mean that coverage artifacts, not grammar topics, drove most of the filtering.

Limitations

• The comparison between AFM and PFA is confounded with estimation paradigm: the AFM implementation uses fixed effects and the PFA implementation uses random effects, so agreement between them is descriptive, not an independent replication.
• AFM and PFA assume linear, additive effects of practice opportunities on performance, which may not capture non-linear second-language acquisition dynamics such as U-shaped learning or interdependencies between linguistic structures.
• The analysis uses only first-attempt responses, isolating independent performance before feedback and leaving later attempts, feedback responsiveness, and within-item learning unexamined.
• Fine-grained and semantic models differ inherently in parameter count, and neither information criteria nor item-blocked cross-validation fully removes the asymmetry between highly expressive item-level models and more abstract representations.
• Rows with null property mappings had to be removed because the automatic annotation tool missed constructs, and the newer tool variant that reduced exclusions was applied after data collection, so semantic-model results remain sensitive to annotation coverage.

Citation

El Hefny, W., & Meurers, D. (2026). A Comparative Study of Knowledge Component Granularity in Intelligent Language Tutoring Systems. Journal of Educational Data Mining, 18(2), 194-241.