Flash Attention 4
The fourth-generation implementation of the Flash Attention series, designed specifically for NVIDIA Hopper and Blackwell architectures. It leverages TMA asynchronous data movement, producer-consumer warp pipelining, and FP8 low-precision path optimizations, achieving forward-pass MFU exceeding 75% of theoretical peak performance.
Timeline (last 90 days)
Flash Attention 4 (FA4) is designed specifically for the NVIDIA Hopper/Blackwell architecture, with core improvements including asynchronous data movement using TMA, pipelined producer-consumer warp separation, and FP8 low-precision path optimization.
On the H100, FA4's forward pass can achieve an MFU of over 75% of the theoretical peak.
All Facts (2)
Flash Attention 4 (FA4) is designed specifically for the NVIDIA Hopper/Blackwell architecture, with core improvements including asynchronous data movement using TMA, pipelined producer-consumer warp separation, and FP8 low-precision path optimization.
50%UnverifiedOn the H100, FA4's forward pass can achieve an MFU of over 75% of the theoretical peak.
50%