Roofline Model
A visual analytical model for assessing the performance ceiling of computational workloads. It identifies computational bottlenecks by examining the relationship between arithmetic intensity (FLOP/Byte) and a hardware platform's peak compute throughput and memory bandwidth.
Timeline (last 90 days)
In batch-1 decoding scenarios, because arithmetic intensity is extremely low, the performance bottleneck must fall on the memory bandwidth side, and the theoretical token ceiling can be calculated from measured bandwidth multiplied by weight bytes.
The Roofline model relates a kernel's arithmetic intensity (floating-point operations per byte of data) to the hardware's memory bandwidth and peak compute, and is used to determine whether to prioritize optimizing memory access or compute logic.
大语言模型自回归生成阶段属于内存带宽受限任务，单batch推理算术强度约1-2 FLOP/Byte，远低于RTX 4090的计算/带宽比约160 FLOP/Byte
单Token生成的算术强度约为1 FLOP/Byte，导致GPU计算单元利用率不到1%
All Facts (4)
In batch-1 decoding scenarios, because arithmetic intensity is extremely low, the performance bottleneck must fall on the memory bandwidth side, and the theoretical token ceiling can be calculated from measured bandwidth multiplied by weight bytes.
50%UnverifiedThe Roofline model relates a kernel's arithmetic intensity (floating-point operations per byte of data) to the hardware's memory bandwidth and peak compute, and is used to determine whether to prioritize optimizing memory access or compute logic.
50%Unverified大语言模型自回归生成阶段属于内存带宽受限任务，单batch推理算术强度约1-2 FLOP/Byte，远低于RTX 4090的计算/带宽比约160 FLOP/Byte
50%Unverified单Token生成的算术强度约为1 FLOP/Byte，导致GPU计算单元利用率不到1%
50%