GEMV
General Matrix-Vector Multiplication, an operation that computes the product of a matrix and a single-column vector. In LLM inference, it applies to the decode phase with a batch size of one and exhibits a linearly predictable memory access pattern.
Timeline (last 90 days)
In Candle's quantized Metal execution path, when the number of rows in the input matrix equals 1 it takes the GEMV path, and when the row count exceeds 1 it enters the GEMM path.
In small-batch scenarios, the additional overhead of the GEMM implementation far outweighs the gains from reusing weights across rows.
GEMV can complete dequantization while reading the weights, whereas GEMM's tiling logic makes the dequantization process more complex.
The author added a fallback mechanism: splitting small-batch inputs into individual rows, running GEMV on each row separately, and then concatenating the results.
At a batch size of 2, the row-by-row GEMV fallback brought throughput back to 16.25 tokens/s, roughly on par with the unbatched baseline of 16.15 tokens/s.
Benchmarks show that when the number of concurrent requests reaches 5, the throughput ratio of row-by-row GEMV over GEMM drops below 100% (90.7%), and GEMM regains the advantage.
The two prefill kernels, GEMM and row-by-row GEMV, produced output that was identical in the first 597 characters under the same prompt and greedy decoding, then began to diverge.
Floating-point addition and multiplication are not associative under the IEEE 754 standard, and the different internal summation orders of GEMM and GEMV produce extremely small differences in activation values for the same input.
All Facts (8)
In Candle's quantized Metal execution path, when the number of rows in the input matrix equals 1 it takes the GEMV path, and when the row count exceeds 1 it enters the GEMM path.
50%UnverifiedIn small-batch scenarios, the additional overhead of the GEMM implementation far outweighs the gains from reusing weights across rows.
50%UnverifiedGEMV can complete dequantization while reading the weights, whereas GEMM's tiling logic makes the dequantization process more complex.
50%UnverifiedThe author added a fallback mechanism: splitting small-batch inputs into individual rows, running GEMV on each row separately, and then concatenating the results.
50%UnverifiedAt a batch size of 2, the row-by-row GEMV fallback brought throughput back to 16.25 tokens/s, roughly on par with the unbatched baseline of 16.15 tokens/s.
50%UnverifiedBenchmarks show that when the number of concurrent requests reaches 5, the throughput ratio of row-by-row GEMV over GEMM drops below 100% (90.7%), and GEMM regains the advantage.
50%UnverifiedThe two prefill kernels, GEMM and row-by-row GEMV, produced output that was identical in the first 597 characters under the same prompt and greedy decoding, then began to diverge.
50%UnverifiedFloating-point addition and multiplication are not associative under the IEEE 754 standard, and the different internal summation orders of GEMM and GEMV produce extremely small differences in activation values for the same input.
50%