bitsandbytes
An open-source library for quantization and acceleration of deep neural networks. It supports 8-bit and 4-bit quantization such as NF4 and is commonly used for large-model inference under low GPU memory.
Timeline (last 90 days)
On an RTX 3060, the developer's hand-written Triton fused W4A16 GEMV kernel targeting batch-1 decoding runs at about 25.5 tok/s, matching the roughly 25.7 tok/s of the bitsandbytes NF4 quantization scheme.
The self-developed W4 kernel's perplexity (probe ppl) is 15.76, better than bitsandbytes NF4's 18.98.
In the same test, the bf16 baseline exceeded the self-developed W4 kernel (25.5 tok/s) and bitsandbytes NF4 (25.7 tok/s) at 31.3 tok/s, reaching 29.8% of the theoretical roofline.
Pascal架构对应CUDA计算能力6.x，而vLLM、bitsandbytes等现代推理框架已将最低支持版本提升至7.0，部分功能要求8.0以上
Mainstream quantization toolchains include llama.cpp (GGUF format) and bitsandbytes.
常见的量化工具包括GPTQ、AWQ和bitsandbytes
flash-attention、xformers、bitsandbytes等高性能加速库通常只提供Linux预编译wheel，在Windows上需要从源码编译
All Facts (9)
Transformers框架原生支持bitsandbytes、GPTQ、AWQ等主流量化方案
90%UnverifiedTransformers库集成了GPTQ、AWQ、bitsandbytes等量化技术和Flash Attention加速方案
90%UnverifiedThe self-developed W4 kernel's perplexity (probe ppl) is 15.76, better than bitsandbytes NF4's 18.98.
50%UnverifiedOn an RTX 3060, the developer's hand-written Triton fused W4A16 GEMV kernel targeting batch-1 decoding runs at about 25.5 tok/s, matching the roughly 25.7 tok/s of the bitsandbytes NF4 quantization scheme.
50%UnverifiedIn the same test, the bf16 baseline exceeded the self-developed W4 kernel (25.5 tok/s) and bitsandbytes NF4 (25.7 tok/s) at 31.3 tok/s, reaching 29.8% of the theoretical roofline.
50%UnverifiedPascal架构对应CUDA计算能力6.x，而vLLM、bitsandbytes等现代推理框架已将最低支持版本提升至7.0，部分功能要求8.0以上
50%UnverifiedMainstream quantization toolchains include llama.cpp (GGUF format) and bitsandbytes.
50%Unverified常见的量化工具包括GPTQ、AWQ和bitsandbytes
50%Unverifiedflash-attention、xformers、bitsandbytes等高性能加速库通常只提供Linux预编译wheel，在Windows上需要从源码编译
50%