BF16
A 16-bit floating-point format that reduces memory usage by half and increases computation speed compared to FP32. It features a wider exponent field and a narrower mantissa field relative to FP16, and is widely used in large model training and inference workloads.
Core Facts
A 70B-parameter FP16 model requires approximately 140 GB of video memory, approximately 70 GB after INT8 quantization, and approximately 35–40 GB after INT4 quantization.
BF16 (Brain Floating Point 16) is a floating-point variant proposed by Google that retains the 8-bit exponent range of FP32 but reduces the mantissa to 7 bits.
Compared to FP16, BF16 has the same 8-bit exponent width as FP32 and therefore a larger dynamic range, making it the mainstream precision choice for current large model training.
Timeline (last 90 days)
BF16 is a 16-bit floating-point format introduced by Google that has a wider exponent range compared to FP16, making it less prone to numerical overflow.
The solution uses BF16 precision uniformly for the KV cache, and MXFP4 quantization for the model weights.
For running high-concurrency vLLM on a server, choose AWQ, GPTQ, or FP8 quantized versions, or the official BF16 original
Inferencing a 1T-parameter MoE model requires hundreds of gigabytes of VRAM even with BF16 half-precision storage, far exceeding the capacity limit of consumer-grade GPUs.
The benchmark performance of IQ3_S is essentially close to the original BF16 precision.
DeepEP compresses Token data from BF16 to FP8 E4M3, halving the volume, and attaches a scale factor to each Token for restoration, which constitutes per-token quantization granularity.
The DeepGEMM-Ascend library supports BF16 (16-bit), FP8 (8-bit), and FP4 (4-bit) numeric formats.
Quantization precision recommendations: prioritize BF16 for production environments, INT4 is acceptable for test environments, and the minimum threshold should not go below INT4, as lower precision results in greater loss of model capability and output quality.
Compute-bound means the hardware's compute units are fully saturated, at which point increasing batch size or optimizing numerical precision (such as using BF16 instead of FP32) becomes meaningful.
The BF16 precision model's baseline accuracy was 74%, which improved to 84% after applying the penalty, with reasoning tokens reduced by 19.4%.
12 more timeline events
All Facts (20)
A 70B-parameter FP16 model requires approximately 140 GB of video memory, approximately 70 GB after INT8 quantization, and approximately 35–40 GB after INT4 quantization.
90%VerifiedBF16 (Brain Floating Point 16) is a floating-point variant proposed by Google that retains the 8-bit exponent range of FP32 but reduces the mantissa to 7 bits.
90%VerifiedCompared to FP16, BF16 has the same 8-bit exponent width as FP32 and therefore a larger dynamic range, making it the mainstream precision choice for current large model training.
70%VerifiedThe core principle of quantization is to convert model parameters from high-precision floating-point representations (such as FP32's 32-bit floating point) to low-precision representations (such as INT8's 8-bit integer or INT4's 4-bit integer).
65%VerifiedBF16最初由Google Brain团队为TPU设计，保留与FP32相同的8位指数位，尾数从23位缩减到7位
65%Verified混合精度训练将权重保存为FP32，而前向和反向传播计算使用FP16或BF16，可将显存占用减少近一半
65%Partially VerifiedBF16 is a 16-bit floating-point format introduced by Google that has a wider exponent range compared to FP16, making it less prone to numerical overflow.
65%UnverifiedThe solution uses BF16 precision uniformly for the KV cache, and MXFP4 quantization for the model weights.
50%UnverifiedFor running high-concurrency vLLM on a server, choose AWQ, GPTQ, or FP8 quantized versions, or the official BF16 original
50%UnverifiedInferencing a 1T-parameter MoE model requires hundreds of gigabytes of VRAM even with BF16 half-precision storage, far exceeding the capacity limit of consumer-grade GPUs.
50%UnverifiedThe benchmark performance of IQ3_S is essentially close to the original BF16 precision.
50%UnverifiedDeepEP compresses Token data from BF16 to FP8 E4M3, halving the volume, and attaches a scale factor to each Token for restoration, which constitutes per-token quantization granularity.
50%UnverifiedThe DeepGEMM-Ascend library supports BF16 (16-bit), FP8 (8-bit), and FP4 (4-bit) numeric formats.
50%UnverifiedQuantization precision recommendations: prioritize BF16 for production environments, INT4 is acceptable for test environments, and the minimum threshold should not go below INT4, as lower precision results in greater loss of model capability and output quality.
50%UnverifiedCompute-bound means the hardware's compute units are fully saturated, at which point increasing batch size or optimizing numerical precision (such as using BF16 instead of FP32) becomes meaningful.
50%UnverifiedThe BF16 precision model's baseline accuracy was 74%, which improved to 84% after applying the penalty, with reasoning tokens reduced by 19.4%.
50%UnverifiedUnder BF16, weight updates occupy 2 bytes per parameter, meaning the raw data volume of a single weight update for a large model with tens of billions of parameters can reach tens of gigabytes.
50%UnverifiedARCv3 is capable of compressing the size of BF16 weight updates by nearly 50%.
50%UnverifiedCompared to FP16, BF16 has a wider exponent field and a narrower mantissa field, which may cause the optimal configuration path for certain GPU kernels to differ from FP16 or FP32 scenarios.
50%UnverifiedIf BF16's supplementary tuning logic shares state or a global cache with the general-purpose path, it may produce difficult-to-reproduce performance anomalies in mixed-precision scenarios or under concurrent requests.
50%