BatchBench: Evaluating Batch Effect and Domain Shift Robustness in Biomedical Images

AI systems face a tricky problem in biomedical image analysis: a model may perform excellently on its training set, but once it encounters images from a new experimental batch or a different data source, performance can drop sharply. This distribution change caused by differences in experimental conditions is called domain shift, and batch effect is the most common form of it. To systematically measure AI models' ability to withstand such interference, researchers have introduced BatchBench—a public leaderboard based on the JUMP-CP dataset that specifically evaluates the robustness of biomedical image models on unseen experimental batches and data sources.

Why batch effects are the "silent killer" of biomedical AI

The acquisition of biomedical images is highly dependent on experimental conditions: microscope models from different labs, staining protocols, culture batches, and even the temperature and humidity of the day can make the same type of cells exhibit vastly different visual characteristics. For deep learning models, these non-biological variations are easily mistaken for meaningful signals, causing model performance to plummet when generalizing across batches.

According to the original article, although domain adaptation methods have made considerable progress in recent years, there is still a significant performance gap for models in domain shift scenarios. This gap is particularly dangerous in applications with extremely high reliability requirements, such as drug discovery and cell phenotype analysis—a model with 95% accuracy on an internal test set may drop below 70% on a different experimental batch.

BatchBench was proposed precisely to turn this "invisible degradation" into a quantifiable, comparable public metric.

Domain Adaptation is an umbrella term for a class of machine learning methods that address domain shift. Its core goal is to maintain good model performance on the target domain when the distributions of the source domain (training data) and target domain (test data) are inconsistent. Common strategies include: aligning feature representations between the source and target domains (e.g., via adversarial training or maximum mean discrepancy constraints), unsupervised or semi-supervised fine-tuning on the target domain, and learning using domain-invariant causal features. In biomedical images, domain adaptation is often used for generalization across batches, labs, and microscopy modalities. However, most existing methods assume that target domain data are available during training (unsupervised domain adaptation), whereas BatchBench's "leave-batch-out" setting is closer to Domain Generalization—the model never sees the target batch during training, which is more challenging than traditional domain adaptation and is a key difficulty in current robustness research.

JUMP-CP: a large-scale cell painting dataset as a testbed

BatchBench is built on JUMP-CP (Joint Undertaking for Morphological Profiling–Cell Painting), one of the largest publicly available biomedical image datasets. This dataset is generated through the Cell Painting assay—an experimental technique that uses multiple fluorescent dyes to simultaneously label different organelles in cells, thereby capturing rich morphological information.

A key characteristic of JUMP-CP is that it naturally contains strong batch effects and domain shift. Because the data are composed of multiple experimental batches and different data sources, it is exactly an ideal stress-test ground for evaluating robustness. Researchers do not need to artificially inject noise; real changes in experimental conditions are already enough to expose model weaknesses.

Evaluation logic of the leaderboard

BatchBench organizes evaluation in the form of a leaderboard. The core idea is to let participating models encounter a portion of experimental batches during training, and then test them on completely unseen batches and data sources. This "leave-batch-out" design directly simulates real deployment scenarios—the data a model encounters after going live is almost never from the same experimental conditions as the training data.

The public nature of the leaderboard also means that models from different teams and architectures can be compared side by side under a unified standard, avoiding the past problem of each team reporting its own metrics with poor reproducibility.

Cell Painting is a high-content imaging assay developed and promoted by the Broad Institute. Its experimental workflow uses six fluorescent dyes to label multiple organelles and structures in cells: nucleus (Hoechst), endoplasmic reticulum (Concanavalin A), nucleolus (SYTO 14), Golgi apparatus and plasma membrane (WGA), mitochondria (MitoTracker), and actin cytoskeleton (Phalloidin). A single imaging run can produce images in five fluorescence channels, covering thousands of morphological features such as cell morphology, texture, and intensity distribution. These features are widely used for gene function annotation, compound toxicity prediction, and phenotypic screening. Because Cell Painting is extremely sensitive to experimental conditions (dye batches, lighting, culture time), image differences between batches often exceed biological differences, making it a natural testing ground for studying batch effects and domain shift. The JUMP-CP collaborative initiative scaled up these characteristics, providing real rather than simulated distribution shift data for methodological evaluation.

Leave-batch-out design is an experimental protocol for evaluating a model's out-of-distribution generalization ability: certain experimental batches or data sources are completely excluded from the training set, the model is trained only on the remaining batches, and then tested on the held-out batches. This is fundamentally different from randomly splitting training/test sets—under random splitting, images from the same batch may appear in both training and test sets, and the model can "memorize" batch-specific visual patterns (such as background tint or illumination gradients), thereby overestimating its generalization ability. The leave-batch-out design forces the model to learn biological signals unrelated to batch, and is closer to real deployment scenarios: data from a new round of experiments inevitably carries technical variation not seen during training. This protocol is also called "leave-one-domain-out" in the domain generalization literature and is one of the gold standards for measuring robustness. BatchBench applies it to the multi-batch structure of JUMP-CP, giving the evaluation results clear practical significance.

Significance for the AI research community

The value of BatchBench goes beyond a single leaderboard. It provides a reproducible benchmark, giving research directions such as domain adaptation and robustness optimization a common frame of reference. For teams working in biomedical image analysis, this leaderboard can answer a very practical question: will my model still work on a different experimental batch?

From a broader perspective, the JUMP Cell Painting leaderboard also echoes the AI field's growing attention to "out-of-distribution generalization." Whether for natural images or medical images, model reliability outside the training distribution is becoming a metric valued more than simply pushing accuracy higher. BatchBench brings this issue into the specific context of biomedical images and provides an actionable evaluation framework.

For researchers who want to participate, BatchBench provides clear submission entry points and evaluation procedures; for adopters, it is a mirror that reveals the true quality of models under the variable conditions of the real world.

One-line Summary

BatchBench is a public leaderboard based on JUMP-CP that systematically evaluates the robustness of biomedical image models on unseen experimental batches.

Paragraph Summary

BatchBench is a public leaderboard for biomedical image analysis that aims to systematically measure AI models' ability to resist domain shift (especially batch effects). It is based on the large-scale JUMP-CP cell painting dataset, which naturally contains strong batch effects and domain shift. The leaderboard uses a leave-batch-out design: models are trained on some experimental batches and tested on completely unseen batches and data sources, directly simulating real deployment scenarios where training and test conditions differ. The article notes that although domain adaptation methods have progressed, the performance gap of models under domain shift remains significant, and this "invisible degradation" is particularly dangerous in drug discovery and cell phenotype analysis. Through unified, reproducible evaluation standards, BatchBench provides a public frame of reference for domain adaptation and robustness research, echoing the AI field's growing attention to out-of-distribution generalization.