Autoregressive Decoding
Timeline (last 90 days)
Frontier API providers price output tokens at 4 to 5 times input tokens because autoregressive decoding is limited by GPU high-bandwidth memory bandwidth.
In autoregressive decoding, the probability of each token is conditioned on all preceding tokens, with the mathematical form P(x_t | x_1, ..., x_{t-1}).
Memory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.
Matrix multiplication is typically compute-bound in high-batch inference scenarios, while autoregressive decoding with a batch size of 1 is almost invariably bandwidth-bound.
A drawback of autoregressive decoding is that inference speed is linearly constrained by sequence length.
The bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.
All Facts (6)
Frontier API providers price output tokens at 4 to 5 times input tokens because autoregressive decoding is limited by GPU high-bandwidth memory bandwidth.
50%UnverifiedIn autoregressive decoding, the probability of each token is conditioned on all preceding tokens, with the mathematical form P(x_t | x_1, ..., x_{t-1}).
50%UnverifiedMemory bandwidth determines generation speed, because autoregressive decoding moves all weights token by token, and the higher the bandwidth, the more tokens generated per second.
50%UnverifiedMatrix multiplication is typically compute-bound in high-batch inference scenarios, while autoregressive decoding with a batch size of 1 is almost invariably bandwidth-bound.
50%UnverifiedA drawback of autoregressive decoding is that inference speed is linearly constrained by sequence length.
50%UnverifiedThe bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.
50%