Swap the Inference Prompt and Rare-Class Recall Comes Back? New Findings on Imbalanced VLM Finetuning

Why Do Rare Classes "Lose Their Memory" Under Imbalanced Finetuning?

After finetuning on class-imbalanced data, vision-language models (VLMs) often exhibit a counterintuitive phenomenon: recall on rare classes can actually drop below the model's own zero-shot performance. In other words, finetuning doesn't just fail to improve the model on these rare classes—it makes it "forget" what it already knew.

A new study centers on this phenomenon and proposes a simple yet effective inference strategy—Cross-Prompt Inference: while finetuning uses a descriptive prompt, at inference time you don't reuse that same descriptive prompt but instead switch to a different descriptive prompt. The results show that this change consistently restores rare-class recall.

What Is the "Matched-Prompt Penalty"?

The researchers call the standard practice of directly reusing the finetuning descriptive prompt at inference the "matched-prompt" setting, and they reveal an unexpected phenomenon they term the matched-prompt penalty: reusing the finetuning prompt can actually yield lower rare-class recall than switching to a different descriptive prompt.

Crucially, this effect does not stem from the quality of the prompt itself, but from the interaction between training and evaluation prompts. The study notes that if the change is not a descriptive prompt change (non-descriptive prompt changes), the effect does not appear. This suggests the core issue lies in the coupling between descriptive semantics and finetuning memory, rather than mere wording quality.

Descriptive prompt refers to a piece of text in natural language that describes the target class or its visual features, such as "a cat with snow-white fur and pointed, upright ears." In zero-shot inference with vision-language models, descriptive prompts are used to construct text embeddings corresponding to each class, which the model then matches against image features. During finetuning, if the training data uses the same set of descriptive prompts as label carriers, the model's parameter updates will form memories around these specific wordings. A so-called "non-descriptive prompt" might be a templated, empty-shell prompt with no class-semantic description (e.g., merely "Class:"), which cannot trigger the same text-image alignment memory and therefore does not induce the same behavioral shift as the matched prompt.

Cross-Model Validation on Six Major Datasets

To verify the generality of cross-prompt inference, the study evaluated multiple different VLMs across six categories of datasets. The conclusion was consistent: cross-prompt inference reliably restores rare-class recall, at the cost of a trade-off of varying magnitude in common-class recall.

The strength of this effect is related to task characteristics—it is strongest when class imbalance and visual confusability jointly cause significant rare-class suppression; when such suppression itself is absent or insignificant, the advantage of cross-prompt inference also weakens or even disappears. This indicates that cross-prompt inference specifically targets "the part of the capability that has been suppressed," rather than being an indiscriminate general-purpose gain.

TuCo Breakdown: The Model Is Pulled Back Toward the Pretrained Side

The researchers further probed the internal mechanism using the Tuning Contribution (TuCo) tool. TuCo decomposes the model's computation into two contributions: the pretrained part and the finetuned part.

The analysis found that cross-prompt inference reduces dependence on the computation introduced by finetuning, pushing the model's computational center of gravity back toward the pretrained components. More direct evidence: when the finetuning contribution is artificially reduced, the advantage brought by cross-prompt inference also shrinks substantially. This supports, at a causal level, the judgment that "rare-class knowledge is actually still retained in the pretrained part."

Tuning Contribution (TuCo) is an analytical toolkit for decomposing the sources of a model's forward computation. Its basic idea is: on the same input, separately compute the contributions of the pretrained weights and the finetuned weights to the final output, approximately decomposing the total computation into a "pretrained part" and a "finetuned part." On vision-language models, this is typically implemented by comparing differences in activations or output logits across layers before and after finetuning. TuCo is not a trainable module but a post-hoc diagnostic method that helps researchers determine whether the model's decisions rely more on new parameters learned during finetuning or on old knowledge already solidified during pretraining. Precisely because of this, it can reveal the phenomenon whereby the computational center of gravity shifts from the finetuned side back to the pretrained side under cross-prompt inference.

A Broader Implication

The value of this work is not just that it provides a zero-cost inference trick that requires no retraining, but more importantly that it redefines the nature of "forgetting" under imbalanced finetuning: the suppressed rare-class knowledge has not disappeared from the model; it is merely hard to invoke under a particular inference prompt.

For practitioners, this offers two lines of thinking. First, after finetuning on imbalanced data, it is worth trying a different descriptive prompt for inference to recover rare-class performance at low cost. Second, excessive reliance on the "finetuning contribution" during finetuning is worth being wary of—over-reliance on the finetuned part may be precisely the root cause of rare-class capabilities being obscured. Moderately steering the model's computation toward pretrained components may be a more robust finetuning path.

One-line Summary

Vision-language models "lose their memory" on rare classes after finetuning on imbalanced data; switching to a different descriptive prompt at inference can restore it.

Paragraph Summary

After vision-language models are finetuned on class-imbalanced data, rare-class recall often falls below zero-shot performance. Researchers attribute this phenomenon to the "matched-prompt penalty": reusing the descriptive prompt used during finetuning at inference time instead suppresses rare-class capabilities. To address this, the paper proposes the cross-prompt inference strategy: still use the original descriptive prompt for finetuning, but switch to a different descriptive prompt at inference time, which reliably restores rare-class recall across six datasets and multiple VLMs, at the cost of a trade-off in common-class recall. Using TuCo analysis, they find that cross-prompt inference reduces dependence on the computation introduced by finetuning and pushes the model's computational center of gravity back toward pretrained components; when the finetuning contribution is artificially reduced, this advantage also disappears, indicating that rare-class knowledge is not lost but merely obscured by a specific prompt.