FP4
A numerical precision format that stores floating-point numbers using 4 bits. Compared with FP16 or BF16, it achieves a fourfold reduction in memory usage at the cost of reduced numerical range and precision. It is commonly used for large model inference optimization.
Core Facts
Timeline (last 90 days)
Low-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.
The success or failure of FP4 training is jointly determined by the scaling contract, operand format, and execution path—it is a systemic engineering problem rather than merely a computational precision problem.
Validation of FP4 schemes must establish evaluation benchmarks covering diverse downstream tasks, and convergence to a lower loss cannot be the sole criterion for scheme selection.
An arXiv paper (arXiv:2610.00053) proposes a method called 'format-aware fusion' to optimize the FP4 pretraining process.
The paper finds that the ranking of downstream tasks is not consistent with the ranking of training loss, and schemes with lower training loss do not necessarily perform better in downstream evaluations.
The bfloat16 baseline throughput is 18.8K tokens/s/GPU, while the fastest custom path reaches 37.9K tokens/s/GPU, nearly doubling the throughput.
The standard FP4 format can only represent a dozen or so distinct values and must be paired with a scale factor mechanism to cover the distribution range of actual model weights.
The DeepGEMM-Ascend library supports BF16 (16-bit), FP8 (8-bit), and FP4 (4-bit) numeric formats.
Under FP4/INT4 quantization, the model size is approximately one-quarter of that under FP16.
When calling GLM 5.3 Flash via OpenRouter, FP4 quantization should be avoided, as FP8 offers the best balance of speed and cost.
6 more timeline events
All Facts (20)
The NVIDIA Blackwell architecture (GB200/B200 series) introduces native FP4 tensor cores at the hardware level
75%UnverifiedGPT-OSS原生支持FP4混合精度，无需额外量化即可大幅压缩显存占用
90%UnverifiedMoRI（Mixture of Routed Inference）量化通信策略在 dispatch 阶段使用 FP4 精度，combine 阶段使用 FP8 精度，实现了 2.56 倍的带宽压缩
90%UnverifiedBlackwell架构支持FP4精度计算，FP4是目前商用AI芯片支持的最低精度格式
80%Unverified许多深度学习模型在权重量化到4位后仍能保持95%以上的推理精度
75%Unverifiedv4.1 Flash's expert weights are shipped natively in FP4 (4 bits per parameter) and no 8-bit version has ever been released; being close to the compression limit of this format, it cannot be further quantized for Agentic tasks.
60%UnverifiedLow-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.
50%UnverifiedThe paper finds that the ranking of downstream tasks is not consistent with the ranking of training loss, and schemes with lower training loss do not necessarily perform better in downstream evaluations.
50%UnverifiedValidation of FP4 schemes must establish evaluation benchmarks covering diverse downstream tasks, and convergence to a lower loss cannot be the sole criterion for scheme selection.
50%UnverifiedThe success or failure of FP4 training is jointly determined by the scaling contract, operand format, and execution path—it is a systemic engineering problem rather than merely a computational precision problem.
50%UnverifiedAn arXiv paper (arXiv:2610.00053) proposes a method called 'format-aware fusion' to optimize the FP4 pretraining process.
50%UnverifiedThe bfloat16 baseline throughput is 18.8K tokens/s/GPU, while the fastest custom path reaches 37.9K tokens/s/GPU, nearly doubling the throughput.
50%UnverifiedThe standard FP4 format can only represent a dozen or so distinct values and must be paired with a scale factor mechanism to cover the distribution range of actual model weights.
50%UnverifiedThe DeepGEMM-Ascend library supports BF16 (16-bit), FP8 (8-bit), and FP4 (4-bit) numeric formats.
50%UnverifiedUnder FP4/INT4 quantization, the model size is approximately one-quarter of that under FP16.
50%UnverifiedWhen calling GLM 5.3 Flash via OpenRouter, FP4 quantization should be avoided, as FP8 offers the best balance of speed and cost.
50%UnverifiedV4.1 Flash uses CSA2 (Compressed Sparse Attention 2) technology combined with FP4 format for caching data, compressing the global KV Cache footprint to only 890 bytes per token.
50%UnverifiedThe FlashVision EXP model has a total parameter count of 304B, uses a MoE architecture, activates only 6 experts during inference, and employs FP4 quantization precision.
50%UnverifiedWhen choosing a quantization format, it is better to choose a slightly larger format that does not trigger CPU offloading (such as FP4) than to blindly pursue the smallest file size.
50%UnverifiedThe maturation of FP4 quantization and tools such as Nunchaku has lowered the hardware barrier for local AI generation, making it feasible to run on 8GB of VRAM.
50%