SageAttention
An efficient attention computation scheme proposed by Tsinghua University and collaborating institutions. It applies a mixed-precision quantization strategy that quantizes Q and K to INT8 while retaining higher precision for V, increasing arithmetic intensity in attention computation. It achieves throughput exceeding FlashAttention 2 and 3 in certain scenarios.
Timeline (last 90 days)
The optimization employs three main techniques: low-precision numerical computation, replacing FlashAttention with SageAttention, and writing custom CUDA kernels.
SageAttention is an efficient attention computation scheme proposed by Tsinghua University and other institutions, with the core idea of applying INT8 quantization to Q and K while retaining higher precision for V.
SageAttention improves arithmetic intensity through mixed-precision quantization, while FlashAttention reduces HBM read/write operations through tiling; the two optimize along different dimensions.
SageAttention relies on the Triton compiler and specific CUDA kernels; because the Triton project long lacked official support for the Windows platform, the success rate of installation on Windows is extremely low.
After upgrading to ComfyUI v0.32.0, users can obtain acceleration effects comparable to SageAttention through a single node, without the need for complex installation and configuration.
In the same test, Sage Attention (Auto) took 14.24 seconds, while PyTorch's default attention took 23.16 seconds.
Both the native attention backend and SageAttention delivered approximately 37% speed improvements compared to the PyTorch default attention implementation.
All Facts (7)
The optimization employs three main techniques: low-precision numerical computation, replacing FlashAttention with SageAttention, and writing custom CUDA kernels.
50%UnverifiedSageAttention is an efficient attention computation scheme proposed by Tsinghua University and other institutions, with the core idea of applying INT8 quantization to Q and K while retaining higher precision for V.
50%UnverifiedSageAttention improves arithmetic intensity through mixed-precision quantization, while FlashAttention reduces HBM read/write operations through tiling; the two optimize along different dimensions.
50%UnverifiedSageAttention relies on the Triton compiler and specific CUDA kernels; because the Triton project long lacked official support for the Windows platform, the success rate of installation on Windows is extremely low.
50%UnverifiedAfter upgrading to ComfyUI v0.32.0, users can obtain acceleration effects comparable to SageAttention through a single node, without the need for complex installation and configuration.
50%UnverifiedIn the same test, Sage Attention (Auto) took 14.24 seconds, while PyTorch's default attention took 23.16 seconds.
50%UnverifiedBoth the native attention backend and SageAttention delivered approximately 37% speed improvements compared to the PyTorch default attention implementation.
50%