[KongchangAI]
Concept

Marlin kernel

A high-performance GEMM implementation in vLLM for low-bit weights. It supports a weight-only quantization path that keeps weights at low bit-width while retaining bf16 activations, and is used to reduce the distribution shift caused by activation quantization.

Timeline (last 90 days)

Nov 7

In the Qwen3.8 27B vLLM tests, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered NVIDIA NVFP4's KLD from 0.0256 to 0.0117, a reduction of more than half.

Unverified50%
Nov 7

In the same test, using the Marlin kernel to keep quantized weights while keeping activations in bf16 lowered Qwen FP8's KLD from 0.0039 to 0.0028, a reduction of about 30%.

Unverified50%

All Facts (2)

Source Articles