Marlin kernel
A high-performance GEMM implementation in vLLM for low-bit weights. It supports a weight-only quantization path that keeps weights at low bit-width while retaining bf16 activations, and is used to reduce the distribution shift caused by activation quantization.
Timeline (last 90 days)
In the Qwen3.8 27B vLLM tests, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered NVIDIA NVFP4's KLD from 0.0256 to 0.0117, a reduction of more than half.
In the same test, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered Qwen FP8's KLD from 0.0039 to 0.0028, a reduction of about 30%.
All Facts (2)
In the Qwen3.8 27B vLLM tests, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered NVIDIA NVFP4's KLD from 0.0256 to 0.0117, a reduction of more than half.
50%UnverifiedIn the same test, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered Qwen FP8's KLD from 0.0039 to 0.0028, a reduction of about 30%.
50%