Intervention Timing at Test Time: How the PORTAL Controller Makes Dynamic Decisions

The Dilemma of Test-Time Intervention: Average Accuracy Masks Drastic Batch-to-Batch Fluctuations

Test-time adaptation allows deployed models to cope with shifts in input distribution, but the same intervention can have opposite effects on different batches. Take updating normalization statistics as an example—it often improves performance on corrupted images, yet once a single class dominates a batch, this same intervention can severely damage accuracy.

The root of the problem lies in how we evaluate: average accuracy conflates success and failure and cannot tell the model "should the intervention be enabled right now." Target labels could serve as a basis for decision-making, but labels are costly and scarce to obtain, and some choices are self-evident and not worth the extra cost of verification.

Test-time Adaptation (TTA) refers to methods in which a model, after deployment and when facing inputs that differ from the training distribution, adjusts itself online using unlabeled or sparsely labeled test data. Common approaches include updating the statistics of normalization layers (e.g., the mean and variance of BatchNorm), entropy minimization, and pseudo-label self-training. The goal of TTA is to let the model adapt to distribution shifts (such as corruption or style changes) without retraining the full model. However, the effectiveness of TTA depends heavily on batch composition: when classes within a batch are balanced, statistic estimates are more reliable; when a batch is dominated by a single class, the updated statistics deviate from the true distribution, causing a sharp drop in performance. This batch-to-batch instability is one of the core risks in the practical deployment of TTA.

PORTAL: Turning "When to Intervene" into a Controllable Decision Problem

To address this contradiction, researchers proposed PORTAL—a controller that decides when to deploy an intervention, which inputs to label, and when to stop querying. Its core idea is to treat every available prediction rule as a candidate and to fix its prediction before the current feedback arrives. In this way, each label obtained amounts to a fair side-by-side evaluation of all candidates on the same input.

PORTAL aggregates shared comparison results from the same sample and directs the query budget toward candidates that could still overtake the current leader. The controller stops when the remaining labels can no longer change the final choice, or when the possible accuracy gain no longer justifies the cost of another query. This mechanism is essentially a trade-off between "sufficiency of evidence" and "cost of querying," rather than blindly continuing to label.

Candidate (prediction rule) in this context refers to a set of optional prediction strategies, such as "no intervention," "update normalization statistics," or "entropy minimization." PORTAL treats each candidate as an independent decision arm and fixes each candidate's prediction for the input before acquiring a new label. Thus, when a label arrives, the same label can simultaneously evaluate whether all candidates are correct or incorrect on that input, forming a "shared comparison." This design draws on the best-arm identification idea from multi-armed bandits: the goal is not to maximize cumulative reward, but to identify the current best candidate with the fewest queries. The key advantage of shared comparison is that the cost of a single annotation is amortized across all candidates, significantly reducing the total number of labels needed to distinguish among them.

How Feedback Requirements Relate to Accuracy Gaps and Shared Errors

The paper's analysis reveals the intrinsic connection between the amount of feedback required and the accuracy gap and shared errors, and explains why some choices require far less evidence than others. When accuracy gaps among candidates are large and the errors they make on different samples are relatively dispersed, a small number of labels suffices to make a reliable judgment; conversely, if candidates perform similarly and their errors overlap heavily, more feedback is needed to distinguish them.

This analysis provides theoretical support for the controller's efficiency: PORTAL does not invest an equal number of labels in every decision, but dynamically adjusts query intensity based on the "distinguishability" among candidates. Experiments confirm this—when a fixed rule already performs well, PORTAL maintains its performance with very little feedback, avoiding unnecessary annotation overhead.

Shared errors refer to the phenomenon in which multiple candidates make mistakes on the same samples simultaneously. When errors among candidates overlap heavily, a single label often cannot distinguish them—because no matter which candidate is chosen, the result on that sample is the same. Conversely, if the error patterns of candidates differ substantially, a single label can provide a strong distinguishing signal. The accuracy gap and shared errors together determine "distinguishability": when the gap is large and errors are dispersed, a few labels suffice to rank candidates reliably; when the gap is small and errors overlap, more feedback is needed. This analysis directly links the query budget to the statistical differences among candidates, providing a theoretical basis for dynamic stopping rules.

Ranking Stability and Preserving Gains Across the Full Corruption Stream

Changing the order in which samples arrive is an easily overlooked risk in test-time adaptation: presenting the same set of data in a different order can lead the intervention strategy to make drastically different choices, triggering severe failures. PORTAL's shared comparison mechanism prevents such drastic fluctuations caused by changes in arrival order and preserves performance gains across the full corruption stream.

Experimental results show that this control mechanism can transform an intervention that "on average loses accuracy" into a beneficial choice on selected batches. In other words, the intervention itself is not necessarily bad—what matters is whether it is applied to the right batch, and PORTAL's value lies precisely in making that judgment.

Periodic Label Checks and the Reuse Value of Feedback

Beyond immediate decisions, PORTAL also introduces a periodic label-checking mechanism to determine whether past evidence has become outdated and to help the controller recover when conditions change. This is especially important in non-stationary environments: the data distribution after model deployment may drift continuously, and evidence that was valid yesterday may not be reliable today.

More noteworthy is the reusability of feedback. Acquired labels are not only used for selection in the current batch but can also be used to train stronger candidates for future batches; and the selection mechanism can bring nearly all of a candidate's accuracy gains into actual deployment. The same feedback thus improves two things at once—the choices made now and the predictive capability available for subsequent batches. This "label once, benefit twice" design allows a limited label budget to deliver compound value.

Non-stationary environment refers to a situation in which the data distribution changes continuously over time, such as seasonal shifts, user behavior drift, or sensor aging. In non-stationary environments, the candidate ranking relationships reflected by previously collected labels may become invalid, and a controller that clings to old evidence will make outdated decisions. Periodic label checks obtain a small number of new labels at regular intervals to verify whether the current leader is still reliable, and trigger re-evaluation when evidence becomes outdated. This is in line with the idea of concept drift detection, but focuses more on maintaining decision quality under a limited label budget rather than merely detecting drift events themselves.

Practical Implications

PORTAL's approach suggests that the core question of test-time adaptation may not be "which intervention to use," but "when to use it, on whom, and to what extent." Modeling intervention decisions as a dynamic comparison among candidates, with labels as a shared evaluation signal, can achieve more robust deployment performance under the real-world constraint of label scarcity. For model systems that need to operate over long periods in distribution-shift environments, this control-level design is just as important as improvements to the intervention methods themselves.