HyperRepair: Restoring the General Capabilities of Fine-Tuned Models at Inference Time

Fine-Tuning's Capability Trade-off

Domain fine-tuning of pretrained language models is a standard way to adapt them to downstream tasks. But researchers have long observed that while fine-tuning improves performance in the target domain, it weakens the model's performance on general tasks. Most existing methods make a trade-off between general capability and domain specialization: either sacrificing some domain performance in exchange for general capability, or accepting a loss in general capability to preserve specialization.

HyperRepair's starting point is to avoid this either/or choice. It aims to preserve domain specialization and restore general capability at the cost of inference alone, without retraining or additional target-domain data.

Generating a Repair Side Network with a Hypernetwork

The core design of HyperRepair is to learn a hypernetwork that learns patterns from the repair process trained on source models. Specifically, the hypernetwork uses activation fingerprints to summarize how a model has changed relative to the pretrained base, and based on this generates a customized Repairing Side Net (RSN).

This RSN carries a prompt-conditioned gating mechanism that can selectively apply corrections. Because the hypernetwork learns the general rules of "how to repair" rather than a patch for a specific model, it can transfer repair capability to unseen models that share the same pretrained base—even when those models come from unseen adaptation domains, and without target-domain samples or additional optimization.

This is the key difference between HyperRepair and per-model repair approaches: repair is no longer a one-off manual process, but is automatically generated by the hypernetwork according to model characteristics.

Hypernetwork is a class of "network that generates network parameters": it takes an input describing a target task or target model and outputs some or all of the weights of another network (the main network). Unlike conventional training where weights are fixed parameters, a hypernetwork's output varies with its input, so the same hypernetwork can dynamically generate customized weights for different models or tasks. In HyperRepair, the hypernetwork does not directly output the repaired model weights but instead generates a lightweight repair side network, thereby compressing the knowledge of "how to repair" into the hypernetwork's parameters. The advantage of this design is generalization: as long as the input can characterize model differences, the hypernetwork can generate a repair network for unseen models without optimizing separately for each model. The cost is that the hypernetwork itself needs sufficient training signal to learn repair patterns, and its output dimensions must match the repair side network's structure.

Activation Fingerprints refer to a compact representation formed from statistics of a model's intermediate-layer activations (such as mean, variance, or dimensionality-reduced vectors) on a set of representative inputs, used to characterize what changes a model has undergone relative to the base model. It is like a model's "behavioral fingerprint": different fine-tuning directions leave different activation shift patterns, so it can serve as a conditioning input to the hypernetwork, letting it determine what kind of correction the model needs. Compared with directly comparing weights, activation fingerprints are insensitive to parameter scale, cheap to compute, and can reflect differences at the functional level; however, they depend on the representativeness of the chosen probe inputs—if the probe set differs too much from the target domain distribution, the fingerprint may be insufficient to distinguish subtle patterns of capability degradation.

Experimental Setup and Quantitative Results

The paper evaluates on two model families, 12 LoRA-adapted datasets, and 15 general benchmarks. Under the default split, HyperRepair achieves 93.4% of the base model's average general score while retaining 99.5% of the fine-tuned model's average domain score.

These two numbers show that restoring general capability does not come at the cost of a significant sacrifice in domain performance. For scenarios that need to serve multiple domain-adapted versions on the same base, this "learn once, repair many" approach has practical appeal, because it compresses repair cost into the inference stage.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method: it freezes the pretrained model's original weights, injects low-rank decomposition matrices (the product of two small matrices) alongside linear layers such as those in Transformer attention, and trains only these newly added parameters. Because the number of new parameters is far smaller than in full fine-tuning, LoRA greatly reduces memory and storage costs and allows multiple adapted versions to share the same base model and be switched on demand. This is precisely the realistic basis for HyperRepair's evaluation scenario: the same base may host a large number of LoRA-adapted versions across different domains, and manually repairing general capability one by one is costly. Note that LoRA's low-rank constraint limits how much it can change the base weights, so its capability degradation patterns may differ from full-parameter fine-tuning; therefore, generalizing the paper's conclusions to full-parameter fine-tuning requires additional validation.

Significance and Limitations of the Method

HyperRepair's value lies in shifting general capability restoration from "retraining" to "inference-time correction." The combination of hypernetwork and activation fingerprints allows repair to generalize to unseen adaptation domains, reducing dependence on target-domain labeled data.

However, the paper's evaluation focuses on LoRA adaptation and two model families, and the method's applicability to full-parameter fine-tuning or other adaptation forms remains to be verified. In addition, the effectiveness of the repair side network depends on whether the repair patterns the hypernetwork learns on source models are general enough, and cross-base-architecture transfer capability also needs more evidence.

Summary

HyperRepair offers a path to restore the general capabilities of fine-tuned models at inference time, using a hypernetwork to generate a gated repair side network and achieving a good balance between general capability and domain specialization across multiple datasets and benchmarks. For teams concerned with model adaptation quality and deployment cost, this direction is worth ongoing attention.

One-line Summary

HyperRepair uses a hypernetwork to generate a repair side network at inference time, restoring the general capabilities weakened in fine-tuned large models.

Paragraph Summary

To address the trade-off in which domain fine-tuning degrades general capability, HyperRepair proposes repairing models at inference time rather than by retraining. At its core, it learns a hypernetwork that uses activation fingerprints to summarize how a model has changed relative to the pretrained base and generates a customized Repairing Side Net (RSN), which selectively applies corrections through a prompt-conditioned gating mechanism. Because the hypernetwork learns general repair patterns, the repair capability can transfer to unseen domain-adapted models sharing the same base, without target-domain data or additional optimization. Experiments cover two model families, 12 LoRA-adapted datasets, and 15 general benchmarks, achieving 93.4% of the base model's average general score while retaining 99.5% of the fine-tuned model's average domain score. The method is currently validated mainly on LoRA adaptation, and its applicability to full-parameter fine-tuning and cross-architecture transfer remains to be tested.