FP8

FP8 quantization technology reduces memory usage by approximately 50% with almost no loss in precision by compressing weights into an 8-bit representation.

FP8 is an 8-bit floating-point format natively supported by newer architectures such as the NVIDIA H100, which can compress data volume by half compared to FP16/BF16

Quantization (such as INT8 and FP8) can reduce memory usage and improve throughput while maintaining acceptable accuracy loss in most scenarios.

FP8 data volume is half that of FP16, and the H100's FP8 Tensor Core compute throughput is twice that of FP16 (approximately 1979 TFLOPS).

The model weights of Flux 2's 9B parameter model occupy approximately 18GB of VRAM at FP16 precision, typically requiring more than 24GB of GPU VRAM.

MoRI（Mixture of Routed Inference）量化通信策略在 dispatch 阶段使用 FP4 精度，combine 阶段使用 FP8 精度，实现了 2.56 倍的带宽压缩

On a dual-Spark system, DeepSeek V4 Flash (FP8 quantization) peaks at approximately 53 tokens per second.

The AWS team used Triton's backend routing mechanism to let different computation paths go to the backend implementations suited to each, thereby maintaining numerical stability under FP8 quantization.

The AWS engineering team carried out kernel-level optimizations on Amazon SageMaker AI for a fused MoE model, including Swap-AB to improve WGMMA utilization, Triton backend routing to implement FP8 quantization, and eliminating repeated Triton JIT compilation.

Low-precision training and inference (FP16, FP8, FP4) are key to the current ability of large models to scale to trillion parameters.

MiniCPM-V 4.7's quantization formats cover GGUF, BNB, AWQ, and GPTQ, and it adds FP8 quantization support, with the GGUF version already uploaded to Hugging Face.

AWQ, GPTQ, and FP8 are formats specifically optimized for CUDA GPUs and cannot run efficiently in a CPU-only environment

For running high-concurrency vLLM on a server, choose AWQ, GPTQ, or FP8 quantized versions, or the official BF16 original

FP8 has a narrower numerical range and lower precision, and correct usage requires fine-grained scaling factor management and numerical stability guarantees at the GEMM kernel level.

OpenDLSS NR's neural network contains 71 blocks, uses FP8 activations, and has total weights of approximately 141 MB.

FP8 is a low-precision data format introduced by NVIDIA in the Hopper and Ada Lovelace architectures, with the Ada architecture's Tensor Cores natively supporting FP8 operations.

With a configuration of 16384 Tokens per card, 7168 values per Token (2 bytes each), and 6 copies, a single Dispatch requires approximately 1.4 GB of data movement per card, and FP8 can halve this amount.

The DeepGEMM-Ascend library supports BF16 (16-bit), FP8 (8-bit), and FP4 (4-bit) numeric formats.

Compared to FP16, FP8 reduces memory footprint and bandwidth consumption by half, and the AIC's compute throughput for FP8 can reach twice that of FP16

EP Buffer supports the FP8 data type and Cache Handle-related operations, and uses UMA to advance communication in the background while freeing AIV cores