The Dataset Quality Dilemma in Digital Pathology AI: Distinguishing Annotation Inconsistency from Domain Shift

A Real-World Computer Vision Challenge

In digital pathology, dataset quality often determines success or failure more than model architecture does. A Reddit post from an engineer working on tumor cell detection in HER2-stained breast tissue laid bare a challenge that's deeply representative of medical AI deployment: when model performance stalls, is the problem in the annotations, the data distribution, or the model itself?

The team was developing an object detection model to detect tumor cells in HER2-stained breast tissue and classify them into four grades: 0, 1+, 2+, and 3+ — the standard grading system used in clinical pathology to assess HER2 expression levels. As new cases from different hospitals or scanners were incorporated in successive "waves," the dataset grew to approximately 4,500 image patches and nearly one million cell annotations.

In terms of class distribution, grade 0 accounts for 28.5%, 1+ for 48.3%, 2+ for 12.3%, and 3+ for 10.9%. This is a classic imbalanced distribution where the minority classes (2+ and 3+) are underrepresented — and these happen to be the most clinically contentious categories requiring the most precise differentiation.

HER2 (Human Epidermal Growth Factor Receptor 2) is one of the most important biomarkers in breast cancer diagnosis and treatment decisions. Pathologists evaluate HER2 protein expression intensity on tumor cell membranes via immunohistochemistry (IHC) staining, scoring results as 0, 1+, 2+, or 3+ per ASCO/CAP guidelines: 0 and 1+ are negative, 3+ is positive (typically qualifying directly for targeted therapy), and 2+ is "equivocal" — requiring further fluorescence in situ hybridization (FISH) testing to determine gene amplification status. This grading result directly affects whether patients can receive HER2-targeted drugs such as trastuzumab (Herceptin), making its clinical stakes enormous. The boundary between 1+ and 2+ is not only a challenge for AI, but a long-standing gray zone among human pathologists. The recent rise of the "HER2-low" concept (1+ or 2+/FISH-negative) has made precise identification of this boundary increasingly critical in clinical practice.

Three Suspects Behind the Performance Bottleneck

The team's best model achieves a macro-average F1 of approximately 0.73 on the combined test set — only slightly above the team's internal acceptance threshold of 0.7. Even more problematic, performance fluctuates significantly across waves and classes. Confusion matrix analysis shows that 21.9% of cells labeled as 2+ are predicted by the model as 1+ — precisely the boundary that's clinically recognized as the hardest to distinguish.

The team suspects inconsistent annotation standards are a major contributing factor. Their current annotation guidelines are roughly 200 words, with no image examples and no rules for handling ambiguous cases. The 1+/2+ boundary is inherently blurry, and systematic differences in interpretation across pathologists are highly likely.

The true complexity lies in confounding: changes in annotators are entangled with changes in hospitals, staining protocols, and scanner equipment. In other words, when a particular wave performs poorly, you can't tell whether it's because a different pathologist annotated it, a different staining reagent was used, or a different scanner was involved. This confounding makes attribution extraordinarily difficult — and is the root cause of many medical AI projects hitting a wall.

Using Minimal Experiments to Separate Annotation Noise from Domain Shift

For the question of "how to distinguish annotation inconsistency, domain shift, and model error with minimal experiments before re-mobilizing pathologists," there are several pathways worth prioritizing.

The key asset here is that the team has images, original annotations, and predictions from two models on the same cohort simultaneously — providing the foundation for cross-analysis.

The first step is inter-model consistency analysis. If two independently trained models make the same systematic errors in certain regions (e.g., both predict 2+ as 1+) while the human annotations consistently disagree, this typically points to noise in the annotations themselves rather than insufficient model capacity. Conversely, if the two models diverge significantly, the issue is more likely one of model generalization.

The second step is wave-stratified error auditing. Break down F1 scores and confusion matrices by wave, and simultaneously record staining, scanner, and annotator metadata for each wave. Although these variables are confounded, look for natural experiments: are there subsets with "same annotator, different scanner" or "same scanner, different annotator"? Even with small sample sizes, these contrasts can begin to isolate variables.

The third step is a small-scale re-annotation experiment. Randomly sample a few hundred cells from the most contested 2+/1+ boundary cases, have 2–3 pathologists independently re-annotate them, and compute inter-annotator agreement (e.g., Cohen's Kappa or Fleiss' Kappa). If human experts themselves can't reach consensus, it's an inevitable consequence that the model can't learn this boundary well — reframing the problem from "the model isn't good enough" to "the label definition is unclear."

Domain Shift refers to the phenomenon where model performance degrades when the training and test data distributions differ. In digital pathology, sources of domain shift are especially diverse: different hospitals use different brands of staining reagents, causing tissue of the same HER2 grade to display dramatically different color saturation and contrast; different models of whole-slide image (WSI) scanners differ systematically in resolution, depth of focus, and color reproduction; and minor variations in tissue fixation and slide preparation affect the textural features of the final image. Combined, these factors mean a model trained on data from a single source often fails to generalize to new hospitals or equipment, even when annotation quality is identical. Domain Adaptation and Stain Normalization are the mainstream technical approaches in academia for addressing this problem — but the prerequisite is isolating domain shift from other noise sources first, to determine whether it warrants dedicated modeling.

Building a Sustainable Annotation Quality Framework

Regarding "what annotation guidelines and QC elements each new wave should have," this is really about upgrading a one-time data cleanup into a sustainable data governance process.

A proper annotation guideline is far more than 200 words of text. The core should include a visual example library for boundary categories: for high-contention boundaries like 1+ vs. 2+, provide a set of canonical and borderline examples confirmed by expert consensus, giving annotators reliable anchors. Written descriptions are often inadequate when faced with microscopy images — visual examples are the hard currency of disambiguation.

It also needs explicit rules for uncertain and unannotatable cells. For example: should ambiguous cells be labeled as "uncertain" or forced into a category? Should cells at the edge of the field of view that are cut off be annotated? Without uniform rules for these edge cases, they inevitably become sources of noise.

Inter-annotator consistency evaluation before each new wave is incorporated into training should be institutionalized. A fixed "gold standard" test set can be established, with each batch of new annotations first run through a consistency check against it. Batches that fail to reach a Kappa threshold are held back from the main training set. This amounts to adding a quality gate to the data pipeline, preventing poor-quality annotations from continuously contaminating the model.

It's also recommended to treat metadata (annotator, hospital, staining batch, scanner model) as first-class citizens in the record. Only when these variables are explicitly tracked will it be possible in the future to truly decompose confounding factors through stratified analysis or domain adaptation methods.

Cohen's Kappa and Fleiss' Kappa are standard statistics for measuring inter-annotator agreement, providing an objective measure of annotation reliability after correcting for the probability of chance agreement. Kappa values are typically interpreted using the Landis & Koch scale: 0.61–0.80 is "substantial agreement," 0.81–1.00 is "almost perfect agreement," and below 0.40 is considered "poor agreement." In pathology annotation contexts, research shows that even experienced pathologists' HER2 scoring Kappa values commonly float between 0.6–0.75, while agreement among junior annotators is often lower. Incorporating Kappa values as a quality gate in the data pipeline means you can quantify "what is the maximum model performance ceiling this batch of labels can support" before model training, avoiding investing limited engineering resources into optimizing a training set where label quality is already the bottleneck.

Broader Lessons for Medical AI Data Governance

The value of this case extends well beyond HER2 grading itself. It reveals a frequently overlooked truth: in highly specialized domains that rely on expert interpretation, the ceiling of annotation quality is the ceiling of model performance. When human experts themselves exhibit systematic disagreement, pursuing more complex network architectures is often a diminishing-returns endeavor.

An F1 of 0.73 hovering near the threshold is less a model problem than an external manifestation of a data definition problem. Locking down the root cause with low-cost consistency experiments first, then investing pathologists' valuable time in a targeted way, is the highest-value approach. For any team building medical or scientific imaging AI systems, establishing annotation consensus, quantifying annotation consistency, and tracking data provenance metadata are more foundational than hyperparameter tuning.