Template Leakage in a Bengali Symptom Diagnosis Benchmark Inflates Data Augmentation Gains

A Neglected Benchmark Flaw: Template Proximity in Symptom Datasets

In Bengali (Bangla) symptom-driven disease prediction research, a small structured benchmark containing only 725 unique patient profiles, covering 85 diseases and 172 binary symptoms, is widely used. This dataset is not only limited in scale and class-imbalanced; more critically, many profiles differ by only one or two symptoms—meaning some test samples are extremely close to training samples in feature space. A recent preprint systematically examines this issue and proposes a leakage-controlled data augmentation and template-aware evaluation framework.

The core question of the study is straightforward: when we apply asymmetric Bernoulli perturbation to augment the training set, how much of the resulting accuracy improvement reflects genuine generalization, and how much is simply the model memorizing test samples that closely resemble training templates?

Experimental Design: A 2×2 Framework Separating Augmentation from Tuning

To cleanly quantify the contribution of data augmentation, the authors designed a 2×2 experimental matrix that fully crosses two factors: "whether to apply Bernoulli perturbation augmentation" and "whether to perform hyperparameter tuning." This design avoids a common confound in prior work—many studies report gains from augmentation and gains from tuning together.

The evaluation protocol uses ten-repeated real-profile holdout and parent-grouped inner cross-validation. The purpose of parent grouping is to ensure that different profile variants of the same disease are not simultaneously assigned to both training and validation sets, thereby reducing the risk of template leakage. As controls, the study also introduces four balancing strategies: exact replication, random oversampling, class weighting, and SMOTEN, ensuring that the effect of Bernoulli perturbation is not simply equivalent to "generating more samples."

The model families covered include Bernoulli Naive Bayes (BNB) and three gradient-boosted tree models: XGBoost, LightGBM, and CatBoost.

Bernoulli perturbation augmentation: In the context of binary features (presence/absence of symptoms), Bernoulli perturbation typically refers to flipping the 0/1 value of a symptom bit with probability p, thereby generating variants of the original profile. The term "asymmetric" means that the probabilities of flipping to 1 and flipping to 0 can differ, to match the clinical characteristics of sparse symptoms and unequal costs of false positives versus false negatives. This augmentation does not change labels and only applies structure-preserving random perturbations to features, so it is regarded as a regularization method.

Parent-grouped cross-validation: When multiple profile variants exist for the same disease, random splitting of training/validation sets may place variants of the same parent group (i.e., profiles from the same source) on both sides, causing validation set "leakage." Parent grouping ensures that all variants of the same parent group can only fall into the same fold, thereby simulating the model's generalization performance when facing entirely new patients.

Augmentation Effects Vary by Model: Tree Models Benefit Substantially, BNB Gains Not Robust

The experimental results show clear model dependence. For Bernoulli Naive Bayes, the accuracy improvement after augmentation is no longer significant after Holm multiple-comparison correction, even though BNB achieves the highest average accuracy among all configurations—97.50%. This number is misleading in itself: BNB is already strong after tuning on the original training set, and augmentation does not bring statistically defensible additional gains.

Gradient-boosted tree models tell a different story. XGBoost, LightGBM, and CatBoost improve by 18.10, 28.45, and 10.09 percentage points, respectively. LightGBM shows the largest increase, approaching thirty percentage points. This indicates that for models that need more samples to delineate decision boundaries, structure-preserving Bernoulli perturbation indeed provides effective regularization and diversity supplementation.

But the authors do not stop at reporting gains. They further audit the reasonableness of standard holdout evaluation using Hamming distance, and find that 91.0% of standard holdout profiles differ from training data by only one symptom bit. In other words, under the conventional evaluation protocol, the vast majority of "test samples" are essentially neighbors of training samples in feature space. This figure directly undermines confidence in the high accuracy reported on this benchmark.

Template-Aware Evaluation: True Performance After Purifying 1-Bit Neighbors

To quantify the contamination of evaluation by template proximity, the study designs a "same-disease 1-bit-purged challenge": removing from the test set those profiles that differ by only one symptom bit from same-disease samples in the training set. Under this stricter evaluation, all models show accuracy drops—as expected, because the purged test set is genuinely harder.

The key point is that XGBoost and LightGBM still retain substantial augmentation gains under this challenge. This indicates that their gains from Bernoulli perturbation are not entirely due to template leakage and include genuine decision-boundary improvement. By contrast, if a model's gains completely disappear after purification, that would mean its "improvement" is essentially just better fitting test samples that are template-adjacent.

In addition, the study evaluates model robustness to missing symptoms and spurious symptoms, as well as the faithfulness of BNB explanations. These two supplementary experiments further characterize the usability of different models in real clinical scenarios—after all, in actual diagnosis, patient-reported symptoms are often incomplete and may even contain noise or false positives.

Hamming distance: On binary symptom vectors, Hamming distance is defined as the number of positions at which two equal-length vectors differ. Here, "differing by one symptom bit" means a Hamming distance of 1. The Hamming distance audit quantifies the proximity of test samples to training templates by computing the Hamming distance between each test profile and its nearest same-disease neighbor in the training set. A smaller distance indicates that the test set is more easily conquered by "memorization" rather than "generalization," and is a direct indicator for assessing benchmark leakage risk.

Implications for Benchmark Evaluation

The value of this work is not to negate data augmentation, but to reveal a more general methodological problem: on small-scale structured medical benchmarks with high template similarity, accuracy figures from standard holdout evaluation may be systematically inflated. The fact that 91.0% of test profiles are within one symptom bit of training data means that any model reporting near-perfect accuracy on this benchmark must additionally clarify whether its evaluation protocol controls for template proximity.

The authors' recommendation is clear: benchmark accuracy should be interpreted together with template proximity metrics. For symptom diagnosis research in low-resource languages such as Bengali, this is especially important—the smaller the dataset and the more imbalanced the classes, the more pronounced the amplification effect of template leakage. Bernoulli perturbation is indeed useful for some model families, but its gains can only be reliably measured within a leakage-controlled evaluation framework.

From a broader perspective, this "leakage control + template awareness" evaluation approach is not limited to Bengali or symptom diagnosis. Any medical AI benchmark based on structured patient profiles, with limited sample size and subtle variants among profiles, may face the same evaluation bias. Incorporating Hamming distance audits and purification challenges into standard evaluation pipelines may be a more meaningful next step than pursuing higher benchmark accuracy.

One-line Summary

91% of test profiles in a Bengali symptom diagnosis benchmark differ from training data by only one symptom; augmentation gains require leakage-controlled evaluation.

Paragraph Summary

This study systematically examines template proximity flaws in a Bengali symptom-driven disease prediction benchmark. The benchmark contains only 725 unique patient profiles, 85 diseases, and 172 binary symptoms, with many profiles differing by only one or two symptoms. The authors design a 2×2 experiment to separate the contributions of Bernoulli perturbation augmentation and hyperparameter tuning, using ten-repeated real-profile holdout and parent-grouped inner cross-validation, and compare exact replication, random oversampling, class weighting, and SMOTEN. Results show that BNB gains after augmentation are not significant after Holm correction (average accuracy 97.50%); XGBoost, LightGBM, and CatBoost improve by 18.10, 28.45, and 10.09 percentage points, respectively. A Hamming distance audit finds that 91.0% of standard holdout test profiles differ from training data by only one symptom bit. Under the same-disease 1-bit-purged challenge, all models decline, but XGBoost and LightGBM retain substantial gains, indicating genuine generalization improvement. The authors recommend interpreting benchmark accuracy jointly with template proximity metrics.