W4A16
A quantization term denoting an inference mode in which weights are 4-bit and activations are 16-bit. It can run on hardware that does not support FP4.
Timeline (last 90 days)
W4A16 quantisation refers to a mixed-precision scheme in which weights are stored in 4-bit and activations remain 16-bit; compared with Q4_K_M it retains a higher-precision activation path, and combined with dedicated CUDA kernels such as marlin it can achieve higher throughput on NVIDIA graphics cards, but VRAM usage is more rigid and kernel support requirements are stricter.
Chord is a set of W4A16 MoE kernels open-sourced by Novita Labs, optimized specifically for the Kimi K2.x series of models.
Compared to full FP16 inference, W4A16 can reduce model VRAM usage by approximately 50%–60%.
In W4A16 quantization, W4 means weights are stored as 4-bit integers, and A16 means activations retain 16-bit floating-point precision, which can compress VRAM usage to approximately one-quarter.
All Facts (4)
In W4A16 quantization, W4 means weights are stored as 4-bit integers, and A16 means activations retain 16-bit floating-point precision, which can compress VRAM usage to approximately one-quarter.
70%UnverifiedW4A16 quantisation refers to a mixed-precision scheme in which weights are stored in 4-bit and activations remain 16-bit; compared with Q4_K_M it retains a higher-precision activation path, and combined with dedicated CUDA kernels such as marlin it can achieve higher throughput on NVIDIA graphics cards, but VRAM usage is more rigid and kernel support requirements are stricter.
50%UnverifiedChord is a set of W4A16 MoE kernels open-sourced by Novita Labs, optimized specifically for the Kimi K2.x series of models.
50%UnverifiedCompared to full FP16 inference, W4A16 can reduce model VRAM usage by approximately 50%–60%.
50%