Arithmetic Intensity
A metric that measures the ratio of computation to memory access, expressed in FLOPs/Byte. It is used to determine whether a computational task is compute-bound or memory-bandwidth-bound, and serves as a core concept in GPU performance analysis.
Timeline (last 90 days)
The arithmetic intensity of the decoding stage in LLM autoregressive generation is extremely low: for a 7B-parameter FP16 model, one forward pass requires loading approximately 14 GB of weights but executes only about 14 GFLOPs of computation, resulting in an arithmetic intensity of approximately 1 FLOP/Byte, making it a bandwidth-bound workload.
自回归推理时batch size通常为1，每个token生成只做一次矩阵-向量乘法，算术强度极低，导致memory-bound问题，单用户场景下GPU利用率往往不到10%
自回归推理是memory-bound（内存带宽受限）问题，因为batch size通常为1时算术强度极低，GPU大部分时间在等待数据搬运
单Token生成的算术强度约为1 FLOP/Byte，导致GPU计算单元利用率不到1%
All Facts (4)
The arithmetic intensity of the decoding stage in LLM autoregressive generation is extremely low: for a 7B-parameter FP16 model, one forward pass requires loading approximately 14 GB of weights but executes only about 14 GFLOPs of computation, resulting in an arithmetic intensity of approximately 1 FLOP/Byte, making it a bandwidth-bound workload.
50%Unverified自回归推理时batch size通常为1，每个token生成只做一次矩阵-向量乘法，算术强度极低，导致memory-bound问题，单用户场景下GPU利用率往往不到10%
50%Unverified自回归推理是memory-bound（内存带宽受限）问题，因为batch size通常为1时算术强度极低，GPU大部分时间在等待数据搬运
50%Unverified单Token生成的算术强度约为1 FLOP/Byte，导致GPU计算单元利用率不到1%
50%