Unverified50% confidenceConceptExact time
W4A16 quantisation refers to a mixed-precision scheme in which weights are stored in 4-bit and activations remain 16-bit; compared with Q4_K_M it retains a higher-precision activation path, and combined with dedicated CUDA kernels such as marlin it can achieve higher throughput on NVIDIA graphics cards, but VRAM usage is more rigid and kernel support requirements are stricter.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn W4A16 quantization, W4 means weights are stored as 4-bit integers, and A16 means activations retain 16-bit floating-point precision, which can compress VRAM usage to approximately one-quarter.75% similarVerifiedQ4_K_M uses mixed precision, storing some critical weights at 6-bit and the remainder at 4-bit, with an overall average of approximately 4.5-bit.75% similarUnverifiedINT4 quantization can further compress the KV Cache to 0.5 bytes, but may require calibration data to ensure accuracy.72% similarVerifiedQLoRA quantizes the base model weights in 4-bit NF4 format, making it possible to fine-tune a 13-billion-parameter model on a single GPU.70% similarVerifiedThe GGUF format supports multiple precision levels such as 4-bit and 8-bit, and can compress the memory footprint of a 7-billion-parameter model from approximately 14 GB (FP16) to approximately 4 GB (Q4_K_M quantization), with inference accuracy loss typically within 1–3%.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1043372API
curl https://kongchang.com/api/v1/knowledge/claims/1043372MCP
get_claim(id=1043372)