LRCC
A method that introduces conditional computation into low-rank compression by training a lightweight router for each Transformer block to dynamically select among nested low-rank paths. This enables per-token differentiated allocation of computational resources, improving inference efficiency and performance of compressed models.
Timeline (last 90 days)
The arXiv paper "LRCC: Generalizing Low-Rank Compression with Conditional Computation" proposes a scheme to improve low-rank compression using conditional computation.
LRCC trains a lightweight router for each Transformer block to dynamically select computation channels per token among nested low-rank paths.
During LRCC training, the low-rank factors remain frozen and only the router is optimized, reducing training cost.
LRCC was evaluated on language modeling and zero-shot downstream tasks using the Llama and Qwen series of models.
Under the same average activated parameter budget, LRCC improves average downstream task accuracy by 7.6 percentage points on Llama-2-7B compared to static methods.
Under matched batch-size-1 decoding latency, LRCC simultaneously improves both perplexity and downstream accuracy on Llama-3.2-1B.
LRCC's performance gains do not rely on specialized custom kernels, lowering the barrier for deployment.
Compared to MoE which adds new expert parameters, LRCC reuses existing low-rank factors, resulting in smaller additional parameter overhead.
As a newly released preprint paper, LRCC's performance on larger-scale models and its combined effects with other compression techniques remain to be validated.
All Facts (9)
As a newly released preprint paper, LRCC's performance on larger-scale models and its combined effects with other compression techniques remain to be validated.
50%UnverifiedLRCC trains a lightweight router for each Transformer block to dynamically select computation channels per token among nested low-rank paths.
50%UnverifiedDuring LRCC training, the low-rank factors remain frozen and only the router is optimized, reducing training cost.
50%UnverifiedLRCC was evaluated on language modeling and zero-shot downstream tasks using the Llama and Qwen series of models.
50%UnverifiedUnder the same average activated parameter budget, LRCC improves average downstream task accuracy by 7.6 percentage points on Llama-2-7B compared to static methods.
50%UnverifiedUnder matched batch-size-1 decoding latency, LRCC simultaneously improves both perplexity and downstream accuracy on Llama-3.2-1B.
50%UnverifiedLRCC's performance gains do not rely on specialized custom kernels, lowering the barrier for deployment.
50%UnverifiedCompared to MoE which adds new expert parameters, LRCC reuses existing low-rank factors, resulting in smaller additional parameter overhead.
50%UnverifiedThe arXiv paper "LRCC: Generalizing Low-Rank Compression with Conditional Computation" proposes a scheme to improve low-rank compression using conditional computation.
50%