LoopVQ
A KV-cache vector quantization method for looped Transformer architectures that jointly encodes the channels of multiple loop steps and fine-tunes the codebook, maintaining model performance at extremely low bit widths (0.5 bits per element).
Timeline (last 90 days)
LoopVQ is a KV cache vector quantization method for recurrent architectures that maintains model performance even at an extremely low precision of 0.5 bits per element.
LoopVQ captures redundancy along the recurrent axis by jointly encoding channels across multiple recurrent steps rather than processing each step independently.
LoopVQ fine-tunes the codebook so that the quantized output aligns with the full-precision model's output, compensating for quantization error.
LoopVQ's experiments are conducted on the Ouro and Huginn models.
When combined with TaSQ, LoopVQ retains most of BF16 precision even at an extremely low bit rate of 0.25 bits per element.
The research team implemented LoopVQ in SGLang.
At a 32K token context, LoopVQ in the SGLang implementation nearly halves decoding latency.
Compared with the BF16 baseline, peak serving throughput improves by 4.3x and 6.1x for the 1.4B and 2.6B Ouro models, respectively.
All Facts (8)
Compared with the BF16 baseline, peak serving throughput improves by 4.3x and 6.1x for the 1.4B and 2.6B Ouro models, respectively.
50%UnverifiedLoopVQ captures redundancy along the recurrent axis by jointly encoding channels across multiple recurrent steps rather than processing each step independently.
50%UnverifiedLoopVQ fine-tunes the codebook so that the quantized output aligns with the full-precision model's output, compensating for quantization error.
50%UnverifiedLoopVQ's experiments are conducted on the Ouro and Huginn models.
50%UnverifiedWhen combined with TaSQ, LoopVQ retains most of BF16 precision even at an extremely low bit rate of 0.25 bits per element.
50%UnverifiedLoopVQ is a KV cache vector quantization method for recurrent architectures that maintains model performance even at an extremely low precision of 0.5 bits per element.
50%UnverifiedAt a 32K token context, LoopVQ in the SGLang implementation nearly halves decoding latency.
50%UnverifiedThe research team implemented LoopVQ in SGLang.
50%