LoRA Ranks Don't Need to Be Uniform: New Model Merging Method Improves Multi-Domain Accuracy

Model merging allows multiple independently fine-tuned expert models to be integrated into a single unified model without retraining. But the conventional approach requires all experts to use the same LoRA rank, which effectively forces the rank choices of different domains to be tied together—even though each domain has its own optimal rank. A new study demonstrates that this coupling is a property of the merging method, not an inherent requirement of merging itself.

Where Does the Uniform Rank Constraint Come From?

In the standard pipeline, all experts are trained at the same LoRA rank, and merging naturally carries that rank over. This seems reasonable, but it overlooks a key fact: different domains have very different needs for low-rank adaptation. Mathematical reasoning may require a higher rank to capture complex patterns, while instruction following may perform well at a lower rank. Forcing uniformity means some experts are compelled to operate under suboptimal configurations.

The research team trained 36 experts across three domains (math, instruction following, multilingual), covering four ranks and three model scales, then compared six merging methods under two settings: uniform rank and each domain's own optimal rank. This experimental design directly tests the implicit assumption that "ranks must be uniform."

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that freezes the original weights of a pretrained model and injects two low-rank matrices A and B alongside the linear layers of the Transformer, adapting to downstream tasks by training these two small matrices. The LoRA "rank" r determines the dimensions of these two matrices (i.e., A is d×r, B is r×d). The smaller r is, the fewer trainable parameters, but expressive capacity is also limited; the larger r is, the stronger the adaptation capability, but it may overfit and introduce more noise during merging. Different tasks have different rank requirements: complex reasoning tasks typically need higher ranks to capture fine-grained patterns, while simple classification or style transfer tasks can achieve optimal performance at low ranks. Therefore, choosing ranks independently for each expert is reasonable, but the traditional merging pipeline defaults to all experts using the same rank, stemming from the fact that merging algorithms (such as averaging, TIES) were designed assuming input adapters have the same structure.

Benefits and Risks of Non-Uniform Ranks

Experimental results show that assigning different ranks to experts causes no loss for average merging and utility-based selection methods, and instead brings up to 1.8 percentage points of multi-domain accuracy improvement. However, conflict-resolution methods perform inconsistently, with losses of up to 4 percentage points. This indicates that non-uniform ranks are not a panacea—the choice of merging algorithm is crucial.

According to the research analysis, this divergence has a simple explanation: the merging process only retains a fixed share of pooled spectral components, and methods that select components by magnitude discard many components that could have reduced loss. Worse, they tend to discard the same expert—the instruction adapter has the smallest singular values and is therefore almost entirely discarded, causing IFEval metrics to drop by 16 to 25 percentage points.

Conflict-resolution merging methods (such as TIES-Merging, DARE, etc.) aim to resolve parameter conflicts when merging multiple fine-tuned models. The core steps of TIES include: Trim (pruning small-magnitude parameters), Elect (electing sign directions), and Merge (merging consistent parts). These methods typically judge importance based on parameter magnitude (absolute value size) and discard or rescale smaller-magnitude parameters. However, when experts use different ranks, the singular value distributions of different experts differ significantly, and selection by magnitude systematically favors experts with larger singular values while ignoring experts with smaller singular values that may be critical for specific tasks (such as instruction-following adapters). This bias leads to certain experts being excessively discarded, causing performance degradation.

Selecting Components by Loss Effect

An alternative is to select components based on their impact on loss rather than by magnitude. Doing so preserves all three experts and keeps accuracy nearly flat across an 8× rank range (fluctuating 0.7 percentage points, compared to TIES method fluctuating 6.7 percentage points at 4B scale). In other words, as long as the merging operator does not select by magnitude, ranks can be chosen independently for each expert.

This finding has direct implications for practitioners: there is no longer a need to sacrifice each domain's optimal configuration for the sake of uniform ranks across all experts. When choosing a merging method, priority should be given to operators based on loss effect rather than spectral magnitude. For scenarios requiring multi-domain capabilities, assigning ranks per expert is not only feasible but can also bring considerable accuracy gains.

Loss effect refers to the degree to which a parameter component contributes to the overall loss function during merging, typically approximated through gradient or Hessian information. Unlike selection by parameter magnitude, selection by loss effect preserves components that are critical for reducing loss even if their magnitude is small. For example, an instruction-following adapter may have small singular values, but its loss impact on the IFEval metric is large, so selection by loss effect can avoid discarding it. In practice, loss effect can be approximated by computing the change in loss caused by removing each component on a validation set, or by using Fisher information-based methods. This approach is slightly more computationally expensive but treats different experts more fairly, especially in non-uniform rank scenarios.

Implications for Model Merging Practice

This study clarifies a long-overlooked constraint: uniform ranks are not a necessary condition for merging but a byproduct of specific merging methods. When using averaging or utility selection, non-uniform ranks are harmless and beneficial; conflict-resolution methods, however, require careful evaluation because they may systematically discard certain experts.

From an engineering perspective, this means optimal ranks can be independently selected for each domain during training and then integrated through an appropriate merging operator. This reduces the compromises imposed by uniform ranks and provides greater freedom in rank configuration for multi-expert systems. Future merging method design should perhaps adopt "not selecting components by magnitude" as a principle to avoid systematic bias against specific experts.

One-line Summary

Research demonstrates that uniform LoRA ranks are a property of merging methods rather than a necessity, and non-uniform ranks can improve multi-domain accuracy.

Paragraph Summary

Model merging typically requires all experts to use the same LoRA rank, but new research shows this coupling is a byproduct of specific merging methods, not an inherent requirement of merging itself. Experiments trained 36 experts of different ranks and scales across math, instruction following, and multilingual domains, comparing six merging methods under uniform and non-uniform ranks. Results show non-uniform ranks cause no loss for averaging and utility selection methods, bringing up to 1.8 percentage points of accuracy improvement; while conflict-resolution methods may lose up to 4 percentage points because their magnitude-based component selection systematically discards instruction adapters, causing IFEval to drop 16-25 percentage points. Selecting components by loss effect preserves all experts, with accuracy fluctuating only 0.7 percentage points across an 8× rank range. The conclusion is that when merging operators do not select by magnitude, ranks can be chosen independently for each expert.