MXFP4
A microscaling floating-point quantization format defined by the Open Compute Project (OCP) that compresses each model parameter to 4 bits. It maintains numerical precision at low bit-widths through intra-group shared scaling factors, and is used to reduce cost and improve efficiency in large-model inference workloads.
Timeline (last 90 days)
The solution uses BF16 precision uniformly for the KV cache, and MXFP4 quantization for the model weights.
In the DeepSeek R1 670B (MXFP4 precision) test, Jalapeño achieved approximately 1.7× throughput per watt (19,641 vs. GB300's 11,781), and end-to-end latency was reduced from 5.99 seconds to 1.65 seconds.
MXFP4 belongs to the MX (Microscaling) series of standards and is a block floating-point quantization specification in which a group of weights shares a single scaling factor.
This optimization focuses on the vLLM inference framework and the MXFP4/NVFP4 quantization paths, involving writing or adjusting dedicated compute kernels.
All Facts (4)
In the DeepSeek R1 670B (MXFP4 precision) test, Jalapeño achieved approximately 1.7× throughput per watt (19,641 vs. GB300's 11,781), and end-to-end latency was reduced from 5.99 seconds to 1.65 seconds.
60%UnverifiedThe solution uses BF16 precision uniformly for the KV cache, and MXFP4 quantization for the model weights.
50%UnverifiedMXFP4 belongs to the MX (Microscaling) series of standards and is a block floating-point quantization specification in which a group of weights shares a single scaling factor.
50%UnverifiedThis optimization focuses on the vLLM inference framework and the MXFP4/NVFP4 quantization paths, involving writing or adjusting dedicated compute kernels.
50%