LIRA
A method for dynamic rank allocation in LoRA based on loss perturbation signals that assesses the sensitivity of each layer through second-order approximation, enabling term-level interpretable online rank reallocation.
Timeline (last 90 days)
LIRA (Loss-Informed Rank Allocation) is an online rank allocation method based on loss perturbation, aimed at making the rank reallocation process interpretable item by item.
Under a second-order approximation, LIRA estimates the magnitude of local loss perturbation caused by recent LoRA factor movements, using this as the score for each layer.
LIRA periodically invokes the DELTA (Displacement-Evaluated Loss-Trajectory Allocation) algorithm for rank reallocation.
Using LIRA does not require modifying the downstream inference or deployment stack.
Experimental results show that LIRA outperforms the uniform LoRA and adaptive-rank baselines reported under this protocol.
Current validation of LIRA is concentrated on medium-scale settings such as RoBERTa-base and GLUE, and its performance on larger models and more task types remains to be seen.
All Facts (6)
LIRA (Loss-Informed Rank Allocation) is an online rank allocation method based on loss perturbation, aimed at making the rank reallocation process interpretable item by item.
50%UnverifiedUnder a second-order approximation, LIRA estimates the magnitude of local loss perturbation caused by recent LoRA factor movements, using this as the score for each layer.
50%UnverifiedLIRA periodically invokes the DELTA (Displacement-Evaluated Loss-Trajectory Allocation) algorithm for rank reallocation.
50%UnverifiedUsing LIRA does not require modifying the downstream inference or deployment stack.
50%UnverifiedExperimental results show that LIRA outperforms the uniform LoRA and adaptive-rank baselines reported under this protocol.
50%UnverifiedCurrent validation of LIRA is concentrated on medium-scale settings such as RoBERTa-base and GLUE, and its performance on larger models and more task types remains to be seen.
50%