Memory Bandwidth
A metric measuring the rate at which data is transferred between memory and the processor, typically expressed in GB/s. It represents the primary performance bottleneck during the decode phase when running large language models locally on a device.
Core Facts
Timeline (last 90 days)
Inference speed (tokens/second) ≈ memory bandwidth ÷ model size
In modern AI hardware design, moving data is often more costly than performing mathematical operations, and memory bandwidth frequently becomes the limiting factor
The author of OpenTPU reports that the speed of token generation is limited by memory bandwidth rather than compute itself
Online real-time inference is typically a low-latency task involving small batches or single requests, where the bottleneck is often memory bandwidth rather than compute.
Memory bandwidth determines weight read speed and affects the latency of generating each token, while floating-point compute (FLOPS) determines matrix operation speed and affects the speed of processing input prompts during the prefill stage.
Memory bandwidth, rather than compute (FLOPS), is the core bottleneck determining local LLM inference speed, because generating each token requires reading nearly all model weights from memory into the compute units
The performance ceiling of large model inference often depends not on the chip's peak compute (FLOPS) but on memory bandwidth.
Mobile SoC memory bandwidth is typically in the range of 50 to 100 GB/s, far below the hundreds of GB/s of data center GPUs.
Memory bandwidth is one of the main bottlenecks for inference latency in large models, where compute units are often waiting for data to load, a phenomenon known as the "memory wall".
When choosing an AI device, one should not look only at the generational number but should focus on substantive specifications such as memory bandwidth and memory capacity.
3 more timeline events
All Facts (14)
Each time a token is generated, the Mac must read the entire model from memory, making memory bandwidth the fundamental upper limit on inference speed.
65%Verified大语言模型的推理过程本质上是内存带宽受限（memory-bandwidth bound）的任务，每生成一个Token都需要将模型的全部权重从存储中读取一遍
65%UnverifiedInference speed (tokens/second) ≈ memory bandwidth ÷ model size
50%UnverifiedIn modern AI hardware design, moving data is often more costly than performing mathematical operations, and memory bandwidth frequently becomes the limiting factor
50%UnverifiedThe author of OpenTPU reports that the speed of token generation is limited by memory bandwidth rather than compute itself
50%UnverifiedOnline real-time inference is typically a low-latency task involving small batches or single requests, where the bottleneck is often memory bandwidth rather than compute.
50%UnverifiedMemory bandwidth determines weight read speed and affects the latency of generating each token, while floating-point compute (FLOPS) determines matrix operation speed and affects the speed of processing input prompts during the prefill stage.
50%UnverifiedMemory bandwidth, rather than compute (FLOPS), is the core bottleneck determining local LLM inference speed, because generating each token requires reading nearly all model weights from memory into the compute units
50%UnverifiedThe performance ceiling of large model inference often depends not on the chip's peak compute (FLOPS) but on memory bandwidth.
50%UnverifiedMobile SoC memory bandwidth is typically in the range of 50 to 100 GB/s, far below the hundreds of GB/s of data center GPUs.
50%UnverifiedMemory bandwidth is one of the main bottlenecks for inference latency in large models, where compute units are often waiting for data to load, a phenomenon known as the "memory wall".
50%UnverifiedWhen choosing an AI device, one should not look only at the generational number but should focus on substantive specifications such as memory bandwidth and memory capacity.
50%UnverifiedThe formula for local AI inference speed is: Token/s ≈ memory bandwidth / amount of data read per token
50%UnverifiedThe formula for local AI inference speed is Token/s ≈ memory bandwidth / data volume to be read per token, and large language models generate each token in an autoregressive manner.
50%