CodeQuant2
A clustering-based lookup table (LUT) quantization framework for low-precision large reasoning models. It suppresses activation outliers through rotation and scaling, adapts codebook sizes, and employs inference-aware calibration strategies to reduce bit width while preserving accuracy.
Timeline (last 90 days)
CodeQuant2 is a clustering-based lookup table (LUT) quantization framework designed specifically for low-precision large reasoning models.
CodeQuant2 mitigates the impact of activation outliers through rotation and scaling operations, making the weight distribution more amenable to quantization.
CodeQuant2 uses clustering methods to represent non-uniform weight distributions, capturing weight characteristics more accurately.
CodeQuant2 adopts adaptive codebook sizes, dynamically adjusting the codebook size according to the quantization difficulty of each layer.
CodeQuant2 improves the strategy for selecting calibration samples to better match the decoding statistics during actual inference, thereby enhancing stability on reasoning tasks.
Combined with a custom LUT CUDA kernel, CodeQuant2 achieves significant accuracy improvements on reasoning benchmarks.
CodeQuant2 achieves up to a 2.85x speedup compared with BF16.
The emergence of CodeQuant2 provides a new approach for the deployment of low-precision large reasoning models and is expected to drive further application of quantization techniques in reasoning scenarios.
All Facts (8)
CodeQuant2 is a clustering-based lookup table (LUT) quantization framework designed specifically for low-precision large reasoning models.
50%UnverifiedCodeQuant2 mitigates the impact of activation outliers through rotation and scaling operations, making the weight distribution more amenable to quantization.
50%UnverifiedCodeQuant2 uses clustering methods to represent non-uniform weight distributions, capturing weight characteristics more accurately.
50%UnverifiedCodeQuant2 adopts adaptive codebook sizes, dynamically adjusting the codebook size according to the quantization difficulty of each layer.
50%UnverifiedCodeQuant2 improves the strategy for selecting calibration samples to better match the decoding statistics during actual inference, thereby enhancing stability on reasoning tasks.
50%UnverifiedCombined with a custom LUT CUDA kernel, CodeQuant2 achieves significant accuracy improvements on reasoning benchmarks.
50%UnverifiedCodeQuant2 achieves up to a 2.85x speedup compared with BF16.
50%UnverifiedThe emergence of CodeQuant2 provides a new approach for the deployment of low-precision large reasoning models and is expected to drive further application of quantization techniques in reasoning scenarios.
50%