One Mastery Threshold Does Not Fit All: The Hidden Pitfalls of Knowledge Tracing Models

Intelligent tutoring systems widely use "mastery thresholds" to determine when a student has met the standard and can stop practicing to move on to the next topic. However, a newly published study points out that the same numerical threshold can lead to vastly different instructional decisions across different Knowledge Tracing (KT) models — meaning educators may be unknowingly imposing unfair learning pathways on different student populations.

The Same Number, Different Meanings

The research team evaluated six knowledge tracing models on four publicly available educational datasets and tested 12 threshold levels ranging from 0.50 to 0.99. The evaluation dimensions included: post-advancement performance, advancement coverage, practice burden, and disparities across student groups with different prior ability levels.

The core finding is that Bayesian Knowledge Tracing (BKT) is relatively insensitive to threshold changes, while neural network models become noticeably more selective as thresholds increase. The reason behind this is not a difference in model "personality," but a fundamental difference in what they estimate. BKT estimates a latent mastery probability, whereas neural models estimate the probability of a correct next response.

In other words, when you set a threshold of 0.80, it represents a certain level of mastery in BKT's view, while in a neural model's view it simply means "the likelihood of answering the next question correctly." The same value of 0.80 simply does not mean the same thing across the two model families.

Bayesian Knowledge Tracing (BKT), proposed by Corbett and Anderson in 1994, is one of the longest-standing knowledge tracing models in educational technology. It models a student's knowledge state as a hidden binary variable — "mastered" or "not mastered" — and describes the learning process through four parameters: initial mastery probability, learning rate (the probability of transitioning from unmastered to mastered after each practice attempt), guess rate (the probability of a correct answer in an unmastered state), and slip rate (the probability of an incorrect answer in a mastered state). The model outputs the posterior probability that a student is in the "mastered" state — a latent estimate of knowledge state.

In contrast, neural network models such as Deep Knowledge Tracing (DKT) do not explicitly model "mastery" as a latent state. Instead, they directly leverage a student's historical response sequence through architectures like recurrent neural networks or Transformers to predict the probability of answering the next question correctly. These two outputs are fundamentally different in meaning: the former is a probabilistic estimate of knowledge state, while the latter is a probabilistic prediction of behavioral outcome. This is the root reason why the same numerical threshold carries different meanings across the two model types.

How Large Are the Threshold Differences?

The study further searched for thresholds that strike a balance among performance, extra practice, and advancement rate across 30 predefined instructional scenarios. The results showed that the optimal "balanced threshold" varied enormously across different models and scenarios.

In half of the tested scenarios, the thresholds selected by neural models and BKT differed by more than 0.10. This gap narrows when instructional goals emphasize "reducing extra practice and allowing more students to advance," but widens when high performance guarantees are the priority. For tutoring systems that directly apply a single "rule-of-thumb" threshold, this kind of deviation is enough to alter the learning pace of hundreds or thousands of students.

Stricter ≠ Fairer

Perhaps the most alarming finding in the study concerns educational equity. Intuitively, raising the mastery threshold might seem like a way to ensure students truly solidify their knowledge, thereby narrowing the gap between stronger and weaker students. But the data tell the opposite story.

Stricter thresholds do not reliably narrow performance gaps; they may instead disproportionately restrict student advancement. The study quantified this effect: in some settings, students with higher prior ability advanced at a rate up to 3.26 times higher than students with lower prior ability. In other words, a uniform high threshold may trap already-struggling students in endless practice cycles while high-achieving students advance freely — the exact opposite of a tutoring system's intended goal of closing gaps.

This finding aligns with the "Matthew Effect" in educational research: in systems lacking personalized intervention, students who already have advantages tend to benefit more from the same rules. In the context of adaptive learning systems, a uniform high threshold is equivalent to measuring students at different starting lines with the same ruler. Students with stronger prior ability are already closer to the threshold and can reach it with fewer practice attempts, while students with weaker prior ability may fall into a cycle of "repeated practice but never reaching the bar" — draining motivation and creating factual inequality of opportunity. The 3.26x gap documented in the study suggests that threshold design is itself a hidden resource allocation mechanism, and fairness metrics must be incorporated into the core of system design rather than treated as an afterthought.

Implications for EdTech Practitioners

The practical implications of this research are straightforward: a mastery threshold is not a "magic number" that can be reused across models.

Whenever the underlying KT model changes (e.g., migrating from BKT to DKT) or when instructional priorities shift (e.g., from "ensuring performance" to "promoting advancement"), thresholds must be recalibrated. More importantly, the quality of a threshold cannot be evaluated on a single metric alone — it should be assessed across four dimensions:

• Performance: Did students who advanced actually master the material?
• Practice: Does the threshold impose unnecessary repetitive burden?
• Advancement: How many students are able to progress smoothly?
• Access: Are there systematic disparities across student ability groups?

For teams building adaptive learning systems, this means you cannot simply take the optimal threshold reported for one model in a paper and apply it directly. Threshold selection is fundamentally an engineering decision deeply tied to the semantic meaning of a model's output, the data distribution, and pedagogical values.

Conclusion

This study elevates what might appear to be a technical detail into a matter of educational equity. Parameters like mastery thresholds are often treated by engineers as tunable hyperparameters to be set casually, yet they directly determine real students' learning experiences and opportunity allocation. As more and more educational products integrate neural network KT models, revisiting and carefully calibrating these thresholds has become more necessary than ever.