Nunchaku
An efficient neural network quantization inference framework developed at MIT, designed specifically for diffusion models. It supports low-precision quantization formats such as FP4, significantly reducing model VRAM requirements and improving inference speed for local deployment on consumer-grade GPUs.
Timeline (last 90 days)
The FP4-based model (svdq-fp4_r32) built on Nunchaku has a size of 6.6GB, fits entirely within 8GB of VRAM without CPU offloading, and reduces the per-step time to 3.4–3.8 seconds.
The maturation of FP4 quantization and tools such as Nunchaku has lowered the hardware barrier for local AI generation, making it feasible to run on 8GB of VRAM.
All Facts (3)
Nunchaku的CUDA内核采用了将两个4-bit值打包到一个8-bit寄存器中的策略进行计算
90%UnverifiedThe FP4-based model (svdq-fp4_r32) built on Nunchaku has a size of 6.6GB, fits entirely within 8GB of VRAM without CPU offloading, and reduces the per-step time to 3.4–3.8 seconds.
50%UnverifiedThe maturation of FP4 quantization and tools such as Nunchaku has lowered the hardware barrier for local AI generation, making it feasible to run on 8GB of VRAM.
50%