Uncertainty Estimation Optimizes Expert Annotation for Diatom Classification

In freshwater ecological monitoring, automated identification of diatoms is a cornerstone of large-scale biological assessment. However, the vast number of diatom species and highly imbalanced class distributions pose enormous challenges for reliable classification. A recent study tackles a 759-class diatom dataset, builds a ResNet-18 baseline model, and explores how calibrated confidence estimation can enable selective classification—thereby reducing expert review workload without sacrificing classification integrity.

Challenges in Diatom Classification and the Baseline Model

Diatoms are important indicator organisms for water quality monitoring, but manual identification is time-consuming and labor-intensive. The research team constructed a diatom microscopy image dataset containing 759 classes and adopted ResNet-18 as the baseline classifier. The initial model exhibited overconfidence: the Expected Calibration Error (ECE) was 0.117, meaning there was a clear discrepancy between the model's predicted probabilities and actual accuracy. Such overconfidence is particularly dangerous in safety-critical ecological assessments, as it may cause rare species to be misclassified without being noticed.

Expected Calibration Error (ECE) is a common metric for measuring the discrepancy between model confidence and true accuracy. It is computed as follows: predicted probabilities are divided into several equal-width bins (e.g., 0-0.1, 0.1-0.2, ...), and for each bin the absolute difference between the "average confidence" and the "actual accuracy" is calculated, then weighted and summed by sample proportion. An ECE of 0 indicates perfect calibration, while larger values indicate the model is more "overconfident" or "underconfident." In this article, ECE=0.117 means the model's probabilities deviate from true accuracy by about 11.7 percentage points on average, which would lead to inaccurate threshold setting when performing risk-sensitive selective classification.

Temperature Scaling Calibration Improves Reliability

To address the overconfidence problem, the study employs post-hoc temperature scaling for calibration, with an optimal temperature parameter T=2.12. After calibration, ECE drops significantly to 0.055, and the model's output confidence is closer to true accuracy. This improvement requires no model retraining—only a single-parameter adjustment substantially boosts reliability, demonstrating the practical value of simple calibration techniques. The calibrated probabilities lay the foundation for subsequent selective classification.

Post-hoc temperature scaling is a parametric calibration method that requires no retraining. Its core idea is to divide the model's output logits vector by a scalar temperature parameter T, then apply softmax to obtain calibrated probabilities: ( p_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)} ). T>1 "softens" the probability distribution, reducing overconfidence; T<1 increases confidence. The optimal T is typically searched on a validation set by minimizing negative log-likelihood (NLL) or ECE. Since it introduces only one parameter and does not change the model architecture, temperature scaling is computationally very cheap and is a commonly preferred first choice for deploying calibration in deep learning—provided the model has already converged in training.

Selective Classification: Intelligent Filtering and Expert Triage

The study applies calibrated probabilities to a triage framework: when model confidence falls below a threshold τ=0.934, the sample is rejected and referred to expert review. At 66.2% coverage, the selective macro F1 score remains at 99.01%. Qualitative analysis shows that the system can effectively identify and defer rare taxa (such as AAMB and AASI) and morphologically ambiguous samples, acting as an intelligent filter. This mechanism prioritizes evaluation metrics for rare species, which is crucial for ecological assessment—missing rare species could obscure signals of ecological degradation.

Selective classification (also known as reject learning or triage framework) allows a model to abstain from prediction when confidence is insufficient, referring the sample to human or secondary systems. Its core evaluation metric is the "coverage-accuracy" trade-off curve: coverage refers to the proportion of samples automatically accepted, and accuracy refers to the classification correctness on those samples. Selective macro F1 is the macro-averaged F1 computed on accepted samples in multi-class settings. Unlike fixed-threshold schemes, coverage is adjustable, allowing users to flexibly set τ according to their expert review budget. In this article, τ=0.934 corresponds to 66.2% coverage, meaning about one-third of samples are triaged to experts, while the automatically accepted portion still achieves a macro F1 of 99.01%, reflecting a balance between high accuracy and operability.

Practical Implications for Ecological Monitoring

This study demonstrates that simple calibration techniques can make automated diatom identification more practical and robust. Through uncertainty estimation-driven selective classification, experts only need to review samples the model is uncertain about, greatly reducing workload. The code, environment configuration, and calibration results have been released in an anonymous repository, supporting reproducible research. Although the study is based on a specific dataset, its methodology can be generalized to other biological classification tasks, providing a reference paradigm for large-scale biological monitoring.

For more technical details, refer to the original paper and the open-source framework torch-uncertainty.

torch-uncertainty is an open-source PyTorch-based library for uncertainty estimation and calibration, providing unified interfaces for methods such as temperature scaling, Monte Carlo dropout, deep ensembles, and quantile regression. It supports classification, regression, and semantic segmentation tasks, and includes built-in calibration evaluation metrics such as ECE, NLL, and Brier score. This article uses the framework to complete temperature scaling and selective classification experiments, making the calibration pipeline standardized and easy to reproduce. For teams looking to introduce uncertainty estimation into biological monitoring pipelines, such frameworks can lower the engineering barrier and avoid implementing calibration algorithms from scratch.

One-line Summary

Temperature scaling calibration reduces ECE from 0.117 to 0.055 for 759-class diatom classification, and selective classification maintains 99.01% macro F1 at 66.2% coverage.

Paragraph Summary

This study addresses class imbalance and model overconfidence in 759-class diatom microscopy image classification. Using ResNet-18 as the baseline, it finds an Expected Calibration Error (ECE) as high as 0.117. After post-hoc temperature scaling (T=2.12) calibration, ECE drops to 0.055, and confidence is closer to true accuracy. The study further applies calibrated probabilities to selective classification: when confidence falls below τ=0.934, samples are rejected and referred to experts. At 66.2% coverage, selective macro F1 still reaches 99.01%, and the system can effectively identify and defer rare taxa and morphologically ambiguous samples. This work demonstrates that simple calibration techniques can significantly improve the practicality and reliability of automated diatom identification, reducing expert review burden while prioritizing evaluation metrics for rare species.