The Scaling Truth of EfficientNet in OCT Retinal Disease Classification

In medical imaging, transfer learning has become the mainstream paradigm for retinal disease classification. A widely assumed premise is that the model scaling laws validated on natural images apply equally to medical images. However, a study on the EfficientNet family for spectral-domain optical coherence tomography (SD-OCT) classification at fixed resolution offers strong counter-evidence to this assumption.

Study Design and Experimental Setup

The study systematically evaluated eight ImageNet-pretrained variants, EfficientNet-B0 through B7, on the Kermany dataset for a four-class task (Normal, CNV, DME, Drusen). All models ran at a fixed 224x224 resolution, with instantiated backbone parameter counts ranging from 4.05M to 64.10M.

The data processing pipeline is noteworthy: original grayscale OCT images were replicated into three RGB channels to match the input requirements of ImageNet-pretrained models. Training used a two-stage strategy—first freezing the backbone to train the classification head, then performing partial fine-tuning. Evaluation metrics covered classification performance, discriminative ability, efficiency, and visualization analysis across multiple dimensions.

This fixed-resolution, unified preprocessing design removes interference from input size variation, making parameter count the core variable of scaling effects.

EfficientNet's compound scaling strategy is the premise for understanding the scaling experiments in this paper. EfficientNet's core innovation is using a set of fixed coefficients (α, β, γ) to simultaneously scale network depth, width, and input resolution, with the constraint α·β²·γ² ≈ 2, so that each version upgrade (e.g., B0→B1) roughly doubles total computation. B0 through B7 are eight versions scaled up step by step according to this rule, with parameter counts increasing from about 4 million to about 66 million. Notably, this scaling rule was determined on ImageNet through neural architecture search and grid search, and its "optimal" configuration is deeply tied to the statistical properties of natural images. This paper fixes all variants at 224×224 resolution, effectively retaining only the depth and width scaling dimensions and cutting off the resolution scaling link—this is precisely the design basis that allows it to examine parameter count effects in isolation.

Core Finding: Accuracy Does Not Rise Monotonically with Scale

Experimental results break the "bigger is better" intuition. Test accuracy goes from 95.93% for B0 to 96.95% for B1, showing non-monotonic variation; B7 reaches 96.73%, not surpassing B1.

On the critical Drusen class, B1 achieves the highest F1 score (0.8780), while B7's Drusen F1 is 0.8720 but has the highest Drusen recall (0.9177). This indicates that larger models make different trade-offs between recall and precision rather than improving comprehensively.

A more striking comparison: the OCT accuracy ranking in this study differs markedly from the reported hierarchy on ImageNet—B7 ranks first on ImageNet, while in this task B7 only shows a "secondary accuracy rebound." The researchers therefore point out that scaling effects depend on the specific task and training protocol.

The Kermany dataset (also called OCT2017) is one of the most commonly used public benchmarks in retinal OCT classification, released by Kermany et al. in 2018 alongside their paper, containing four classes of SD-OCT images—Normal, CNV, DME, Drusen—totaling about 80,000 images. A frequently discussed issue with this dataset is that images come from a small number of patients, and in the official split, different slices from the same patient may be scattered across training and test sets—the so-called "cross-partition duplicate samples." If images from the same patient appear in both training and test sets, the model may achieve inflated accuracy by memorizing patient-specific anatomical features rather than pathological features. This corresponds exactly to the limitation candidly acknowledged by the authors—their results are conditional on the data split used, and cross-partition leakage has not been independently audited. Therefore, the 95%+ accuracy in the paper should be understood as a relative comparison under this split protocol, not a measure of absolute generalization ability.

Efficiency Cost and Class Difficulty Distribution

The ratio of accuracy to parameter count drops sharply from 19.82 to 1.48 (percentage points per million total parameters), meaning the marginal benefit of model size expansion decays rapidly. Although B7 shows a secondary accuracy rebound, the cost is higher CPU latency.

Class-level analysis reveals another pattern: Drusen has the lowest F1 across all variants and contributes 82.2% of the total sum of squared class F1 deviations. The class difficulty ordering remains consistently Normal > CNV > DME > Drusen. This stable ordering suggests that the visual features of certain pathology types are inherently harder for current architectures to capture fully, and model scaling cannot change this fundamental fact.

Another noteworthy observation: the final training-test gap has only a weak and non-significant correlation with test accuracy (r = 0.26, p = 0.53). This weakens the simple explanation that "large generalization gap causes poor performance."

The total sum of squared class F1 deviations is the metric used in this paper to quantify "which class concentrates the difficulty." Its calculation is roughly: for each class, compute the squared deviation of each model's F1 score on that class from its mean, then sum over all models; each of the four classes gets a total, and then the percentage of each class in the total is calculated. If one class contributes the vast majority of the deviation, it means model size changes mainly affect that class's performance, while other classes remain relatively stable. In this paper, Drusen contributes 82.2%, meaning that performance fluctuations from scaling are almost entirely concentrated on Drusen, while Normal, CNV, and DME performance is insensitive to model size. The value of this metric lies in decomposing the scalar "overall accuracy" into class-level sources of variance, thereby revealing where scaling gains actually land, rather than merely comparing averages.

Implications for Medical Imaging Practice

This study conveys several pragmatic signals. For SD-OCT classification tasks, the medium-scale EfficientNet-B1 performs best in both accuracy and Drusen F1, making it the most cost-effective choice. Blindly pursuing larger models not only yields limited benefits but also increases inference latency.

The researchers also candidly acknowledge limitations: because cross-partition duplicate samples have not been independently audited, results are conditional on the data split used. This reminds practitioners to pay attention to the reliability of data splits when reproducing or deploying.

More broadly, scaling laws from natural images cannot be directly transferred to medical imaging transfer learning scenarios. For medical image tasks at fixed resolution, the optimal model size needs independent validation rather than following ImageNet's empirical ranking.

One-line Summary

At fixed resolution, EfficientNet scaling effects in OCT classification are non-monotonic, with B1 outperforming B7.

Paragraph Summary

This study systematically evaluates eight ImageNet-pretrained EfficientNet variants, B0 through B7, on the Kermany SD-OCT dataset for four-class classification, with all models fixed at 224×224 resolution to remove input size interference and examine parameter count effects in isolation. Results show that accuracy does not rise monotonically with scale: B1 leads at 96.95%, B7 only reaches 96.73%, and the OCT ranking clearly diverges from the ImageNet hierarchy. The accuracy-to-parameter ratio drops sharply from 19.82 to 1.48, indicating rapidly diminishing marginal returns. At the class level, Drusen has the lowest F1 across all variants and contributes 82.2% of the total sum of squared class F1 deviations, with a stable difficulty ordering of Normal > CNV > DME > Drusen. The training-test gap correlates only weakly with accuracy. The authors therefore argue that scaling laws from natural images cannot be directly transferred to medical imaging transfer learning, and optimal model size requires independent validation.