FP16
A 16-bit half-precision floating-point format and a commonly used numerical precision standard in deep learning. Compared with FP32 it halves memory usage, while retaining a higher dynamic range than quantized formats such as INT8.
Core Facts
In FP16 precision, large models require approximately 2 GB of VRAM per 1 billion parameters.
Compared to FP16, BF16 has the same 8-bit exponent width as FP32 and therefore a larger dynamic range, making it the mainstream precision choice for current large model training.
Timeline (last 90 days)
Quantization techniques can compress trained neural network weights from FP16/FP32 to INT8/INT4, significantly reducing model size, lowering memory usage, and improving inference speed, at the cost of possible accuracy loss.
The KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.
W8A8 quantization means both weights and activations are compressed to 8-bit integers; compared with FP16 it can reduce memory usage and improve NPU throughput, at the cost of possibly introducing slight accuracy loss.
Low-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.
Compared with FP16, 4-bit quantisation can reduce model size and memory usage to about a quarter, which is the key prerequisite for ordinary CPU machines to run local large models.
A 501B-parameter model requires approximately 1TB of VRAM for its full weights in FP16 format.
4-bit quantization means each parameter occupies only 0.5 bytes, reducing VRAM usage by 75% compared to FP16.
Loading a model with N parameters at FP16 precision requires approximately 2N bytes of video memory.
Compared to FP16, FP8 reduces memory footprint and bandwidth consumption by half, and the AIC's compute throughput for FP8 can reach twice that of FP16
Under FP4/INT4 quantization, the model size is approximately one-quarter of that under FP16.
6 more timeline events
All Facts (20)
In FP16 precision, large models require approximately 2 GB of VRAM per 1 billion parameters.
80%VerifiedCompared to FP16, BF16 has the same 8-bit exponent width as FP32 and therefore a larger dynamic range, making it the mainstream precision choice for current large model training.
70%UnverifiedFP16格式下，大模型显存需求的估算公式为：参数量 × 2 = 所需显存（GB），即每个参数以16位（2字节）浮点数存储
95%UnverifiedA 7B-parameter model requires approximately 14 GB of VRAM for model weights alone in FP16 format, and the total VRAM requirement for training may balloon to 50–80 GB.
60%UnverifiedThe perplexity loss of Q4_K_M quantization compared to FP16 is typically within 1%–3%
60%UnverifiedQuantization techniques can compress trained neural network weights from FP16/FP32 to INT8/INT4, significantly reducing model size, lowering memory usage, and improving inference speed, at the cost of possible accuracy loss.
50%UnverifiedThe KV cache size can be expressed with the formula cache = 2 (K and V) × layers × KV heads × head dim × tokens × 2 bytes, where 2 bytes corresponds to FP16, doubling when using FP32, and halving when using INT8 quantization.
50%UnverifiedW8A8 quantization means both weights and activations are compressed to 8-bit integers; compared with FP16 it can reduce memory usage and improve NPU throughput, at the cost of possibly introducing slight accuracy loss.
50%UnverifiedLow-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.
50%UnverifiedCompared with FP16, 4-bit quantisation can reduce model size and memory usage to about a quarter, which is the key prerequisite for ordinary CPU machines to run local large models.
50%UnverifiedA 501B-parameter model requires approximately 1TB of VRAM for its full weights in FP16 format.
50%Unverified4-bit quantization means each parameter occupies only 0.5 bytes, reducing VRAM usage by 75% compared to FP16.
50%UnverifiedLoading a model with N parameters at FP16 precision requires approximately 2N bytes of video memory.
50%UnverifiedCompared to FP16, FP8 reduces memory footprint and bandwidth consumption by half, and the AIC's compute throughput for FP8 can reach twice that of FP16
50%UnverifiedUnder FP4/INT4 quantization, the model size is approximately one-quarter of that under FP16.
50%UnverifiedThe experiment was conducted on a nine-core FP16 GEMM (half-precision general matrix multiplication) test set.
50%UnverifiedCompared to FP16, BF16 has a wider exponent field and a narrower mantissa field, which may cause the optimal configuration path for certain GPU kernels to differ from FP16 or FP32 scenarios.
50%UnverifiedCompared to full FP16 inference, W4A16 can reduce model VRAM usage by approximately 50%–60%.
50%Unverified在Stable Diffusion中FP16精度已成为默认选择，因为半精度浮点数带来的数值误差几乎不影响视觉质量，却能将显存占用直接减半
50%Unverified2-bit量化意味着每个权重仅用4种状态表示，相比FP16压缩比高达8倍
50%