VRAM Bandwidth
The rate of data transfer between video memory and compute units; it directly affects the speed of weight reads during the decoding phase of large models and the resulting inference latency.
Timeline (last 90 days)
Memory capacity determines whether the parameter count plus the KV Cache can fit into the device; memory bandwidth determines the speed of weight reads during each token generation in the decoding phase, and thus directly determines inference latency.
At small batch sizes the bottleneck is memory bandwidth, while at large batch sizes the rising compute density makes compute the bottleneck, which explains the performance crossover between the 3090 and the L40S.
The autoregressive generation phase is a memory-bandwidth-sensitive workload: generating each token requires reading the model weights in full once (or at least reading the weights involved in the activation path), so the actual speed is approximately equal to the VRAM bandwidth divided by the weight volume that needs to be read per token.
The upper limit of decode speed is determined by the ratio of VRAM bandwidth to model weight size, and has little to do with GPU floating-point compute power.
Under AI workloads, the bottleneck is usually VRAM bandwidth and capacity rather than temperature, unless overheating causes throttling.
All Facts (5)
Memory capacity determines whether the parameter count plus the KV Cache can fit into the device; memory bandwidth determines the speed of weight reads during each token generation in the decoding phase, and thus directly determines inference latency.
50%UnverifiedAt small batch sizes the bottleneck is memory bandwidth, while at large batch sizes the rising compute density makes compute the bottleneck, which explains the performance crossover between the 3090 and the L40S.
50%UnverifiedThe autoregressive generation phase is a memory-bandwidth-sensitive workload: generating each token requires reading the model weights in full once (or at least reading the weights involved in the activation path), so the actual speed is approximately equal to the VRAM bandwidth divided by the weight volume that needs to be read per token.
50%UnverifiedThe upper limit of decode speed is determined by the ratio of VRAM bandwidth to model weight size, and has little to do with GPU floating-point compute power.
50%UnverifiedUnder AI workloads, the bottleneck is usually VRAM bandwidth and capacity rather than temperature, unless overheating causes throttling.
50%