FuTransformer
A CUDA implementation of Llama-style Transformer inference optimized for NVIDIA Hopper GPUs. It employs an FP8 quantization path for the prefill stage and an INT4 quantization path for the decode stage, and fuses GEMM with the epilogue to reduce data movement.
Timeline (last 90 days)
FuTransformer is a CUDA implementation targeting NVIDIA Hopper GPUs, with a separate FP8 prefill path and INT4 decode path designed for Llama-style Transformer blocks.
On a single H100, FuTransformer's prefill path is up to 1.48x faster than compiled PyTorch FP8, reaching 1.29 PFLOP/s.
On the gate/up projection with SwiGLU at 8B shapes, FuTransformer's BF16 path is 1.13x faster than CODA, and its FP8 path is 2.02 to 2.18x faster than the evaluated BF16 CODA path.
Across 8B and 70B shapes and batch sizes from 1 to 32, FuTransformer's INT4 decode is 1.3 to 2.4x faster than the faster of PyTorch's BF16 and INT4 blocks.
FuTransformer's INT4 decode is 31% to 51% faster than its own FP8 path.
Full Llama-3.1-8B decodes in 2.8 milliseconds per step at batch 1 using FuTransformer, compared with 8.1 milliseconds for the compiled transformers implementation.
The perplexity difference between FuTransformer's kernel implementation and the quantized reference computation is within 0.001, indicating that the acceleration introduces no additional numerical deviation.
The above acceleration figures all come from a single-GPU H100 experimental environment, and performance under multi-GPU parallelism, longer contexts, and different batch distributions still requires further validation.
All Facts (8)
FuTransformer is a CUDA implementation targeting NVIDIA Hopper GPUs, with a separate FP8 prefill path and INT4 decode path designed for Llama-style Transformer blocks.
50%UnverifiedOn a single H100, FuTransformer's prefill path is up to 1.48x faster than compiled PyTorch FP8, reaching 1.29 PFLOP/s.
50%UnverifiedOn the gate/up projection with SwiGLU at 8B shapes, FuTransformer's BF16 path is 1.13x faster than CODA, and its FP8 path is 2.02 to 2.18x faster than the evaluated BF16 CODA path.
50%UnverifiedAcross 8B and 70B shapes and batch sizes from 1 to 32, FuTransformer's INT4 decode is 1.3 to 2.4x faster than the faster of PyTorch's BF16 and INT4 blocks.
50%UnverifiedFuTransformer's INT4 decode is 31% to 51% faster than its own FP8 path.
50%UnverifiedFull Llama-3.1-8B decodes in 2.8 milliseconds per step at batch 1 using FuTransformer, compared with 8.1 milliseconds for the compiled transformers implementation.
50%UnverifiedThe perplexity difference between FuTransformer's kernel implementation and the quantized reference computation is within 0.001, indicating that the acceleration introduces no additional numerical deviation.
50%UnverifiedThe above acceleration figures all come from a single-GPU H100 experimental environment, and performance under multi-GPU parallelism, longer contexts, and different batch distributions still requires further validation.
50%