RADC: Risk-Aware Dual Caching for Test-Time Adaptation in Vision-Language Models

Why Test-Time Adaptation Breaks Down

Vision-language models like CLIP face a fundamental challenge in real-world deployment: the distribution of training data rarely matches the test environment. To dynamically adapt during inference, researchers have developed Test-Time Adaptation (TTA) techniques, with cache-based methods gaining particular traction due to their low overhead and no need for backpropagation.

However, these methods suffer from two long-overlooked pain points. The first is background bias: CLIP's global representations encode background information along with the target object, so when the subject and background are entangled, global prototypes become noisy and classification degrades. The second is unreliable cache admission: most existing methods use entropy to decide which samples enter the cache, but entropy poorly reflects true sample quality under representation drift — low-quality samples can easily slip in and corrupt subsequent predictions.

This paper (arXiv:2610.06932), published on arXiv, directly addresses both issues with a new framework called RADC (Risk-Aware Dual Caching).

Cache-based TTA methods maintain a dynamic "sample bank" during inference: test sample features and their predicted labels are stored in the cache, and nearest-neighbor retrieval assists subsequent predictions — all without updating model weights. This stands in sharp contrast to gradient-based TTA approaches (e.g., TTT, TENT), making cache-based methods far more practical under compute constraints. CLIP is a popular backbone for this line of research precisely because its text-image pre-training enables strong zero-shot transfer — but its global image encoder treats foreground and background equally, making it especially vulnerable to background noise under domain shift. This is the core motivation behind RADC.

RADC's Core Design: Dual Caching + Risk-Aware Admission

RADC's approach can be summarized as "enhancing prototype learning with a more reliable dual-cache mechanism." It consists of two key components, each targeting one of the pain points described above.

Semantic Foreground Cache: Stripping Away Background Noise

The first component is the Semantic Foreground Cache. Rather than using CLIP's global representation directly, RADC aggregates spatially consistent, class-aligned evidence from CLIP's spatial representations to derive foreground prototypes.

The benefit is that foreground prototypes focus on regions that are genuinely class-relevant, effectively suppressing background interference. This doesn't replace the global cache — it complements it. The global cache preserves overall context while the foreground cache provides cleaner target semantics. Together, they yield a more complete and robust characterization of image content.

CLIP's visual encoder (typically a Vision Transformer) produces not just a global [CLS] token but also patch tokens for each spatial position, carrying local semantic information. RADC's "aggregation of class-consistent spatial evidence" selectively weights these patch tokens: only patches semantically aligned with the predicted class are retained, while background patches are suppressed, and the result is pooled into a foreground prototype. This aligns with the emerging trend of "local feature alignment" in few-shot learning and open-vocabulary detection — essentially pushing CLIP from image-level matching toward region-level matching, achieving implicit focus on target regions without extra annotations or detectors.

Gaussian Risk Admission: Filtering Reliable Samples via Uncertainty

The second component is Gaussian Risk Admission, which governs which samples enter either cache.

The limitation of traditional entropy thresholds is that they only capture the sharpness of a single prediction distribution, ignoring fluctuations in the representation itself. RADC instead models multi-view representations as diagonal Gaussian distributions, jointly considering two dimensions: class separability (whether a sample clearly belongs to a given class) and feature uncertainty (how stable the representation is). By evaluating both, the system prioritizes genuinely reliable cache candidates rather than being misled by superficially high confidence.

This shift from "entropy-driven" to "risk-driven" admission is what fundamentally distinguishes RADC from prior cache-based TTA methods.

Modeling feature representations as Gaussian distributions is a classic approach in Bayesian deep learning. A "diagonal Gaussian" uses a mean vector and a diagonal covariance matrix (independent variance per dimension), drastically reducing parameters compared to a full covariance matrix. RADC's multi-view setup typically applies different data augmentations to the same image to produce multiple feature vectors, which are then fit to a Gaussian — the mean captures expected semantics, while the variance captures inconsistency across augmented views, i.e., feature uncertainty. Class separability can be measured via Mahalanobis distance or inter-class margin relative to class prototypes. Together, these form a "risk" metric that more comprehensively reflects the benefit-versus-risk of admitting a given sample into the cache than a single entropy value ever could.

Inference: Three-Way Prediction Fusion

At inference time, RADC doesn't rely on a single signal — it fuses three complementary predictions:

• Zero-shot logits: CLIP's native zero-shot predictions, providing a general prior;
• Global cache predictions: judgments based on global prototypes, preserving overall context;
• Foreground cache predictions: judgments based on foreground prototypes, emphasizing target semantics.

The three sources complement each other, ensuring that final predictions are grounded in pre-trained knowledge while also benefiting from dynamically accumulated cache information. This multi-path ensemble design proves especially robust when representations shift.

Experimental Results and Significance

The paper conducts extensive experiments on both cross-domain and out-of-distribution (OOD) benchmarks, with RADC achieving consistent state-of-the-art performance across the board. This demonstrates that the combination of dual caching and risk-aware admission is broadly effective at handling real-world distribution shifts.

From a broader perspective, RADC represents a noteworthy direction in TTA research: rather than settling for global representations, it distills cleaner semantic prototypes from spatial evidence; rather than relying on heuristic entropy thresholds for admission, it introduces probabilistic uncertainty modeling. Both ideas offer meaningful inspiration for improving the deployment robustness of vision-language models.

It should be noted that, as a preprint, RADC's additional computational overhead, compatibility with different backbone architectures, and performance under larger-scale or more extreme distribution shifts all await further validation by the community. Nevertheless, its "risk-aware dual caching" paradigm clearly offers a principled and extensible new framework for cache-based TTA.