GEMM
General Matrix Multiplication, the fundamental and computationally intensive operation in deep learning. It is the core computation underlying fully connected layers, attention mechanism Q/K/V projections, and Transformer feed-forward networks, with its execution efficiency directly determining overall model training and inference throughput.
Timeline (last 90 days)
Matrix multiplication satisfies (A×B)ᵀ = Bᵀ×Aᵀ; swapping the two operands and transposing the result does not change the computational semantics, only the way data is arranged in memory and registers.
In Candle's quantized Metal execution path, when the number of rows in the input matrix equals 1 it takes the GEMV path, and when the row count exceeds 1 it enters the GEMM path.
In small-batch scenarios, the additional overhead of the GEMM implementation far outweighs the gains from reusing weights across rows.
Benchmarks show that when the number of concurrent requests reaches 5, the throughput ratio of row-by-row GEMV over GEMM drops below 100% (90.7%), and GEMM regains the advantage.
Floating-point addition and multiplication are not associative under the IEEE 754 standard, and the different internal summation orders of GEMM and GEMV produce extremely small differences in activation values for the same input.
The two prefill kernels, GEMM and row-by-row GEMV, produced output that was identical in the first 597 characters under the same prompt and greedy decoding, then began to diverge.
GEMV can complete dequantization while reading the weights, whereas GEMM's tiling logic makes the dequantization process more complex.
The computational form of GEMM can be described as C = A × B + C, where A, B, and C are all two-dimensional matrices.
BLAS (Basic Linear Algebra Subprograms) is the foundational building block of deep learning and scientific computing, with neural network training and inference relying on large-scale matrix multiplication (GEMM).
In large models based on the Transformer architecture, the vast majority of floating-point operations are concentrated in matrix multiplication, and the efficiency of GEMM kernels directly determines training throughput and inference latency.
5 more timeline events
All Facts (16)
通过CUDA Tile，开发者可以更自然地表达矩阵分块乘法（Tiled GEMM），而不必深入处理wmma或mma.sync等底层PTX指令
85%UnverifiedMatrix multiplication satisfies (A×B)ᵀ = Bᵀ×Aᵀ; swapping the two operands and transposing the result does not change the computational semantics, only the way data is arranged in memory and registers.
50%UnverifiedFloating-point addition and multiplication are not associative under the IEEE 754 standard, and the different internal summation orders of GEMM and GEMV produce extremely small differences in activation values for the same input.
50%UnverifiedIn Candle's quantized Metal execution path, when the number of rows in the input matrix equals 1 it takes the GEMV path, and when the row count exceeds 1 it enters the GEMM path.
50%UnverifiedIn small-batch scenarios, the additional overhead of the GEMM implementation far outweighs the gains from reusing weights across rows.
50%UnverifiedGEMV can complete dequantization while reading the weights, whereas GEMM's tiling logic makes the dequantization process more complex.
50%UnverifiedBenchmarks show that when the number of concurrent requests reaches 5, the throughput ratio of row-by-row GEMV over GEMM drops below 100% (90.7%), and GEMM regains the advantage.
50%UnverifiedThe two prefill kernels, GEMM and row-by-row GEMV, produced output that was identical in the first 597 characters under the same prompt and greedy decoding, then began to diverge.
50%UnverifiedThe computational form of GEMM can be described as C = A × B + C, where A, B, and C are all two-dimensional matrices.
50%UnverifiedBLAS (Basic Linear Algebra Subprograms) is the foundational building block of deep learning and scientific computing, with neural network training and inference relying on large-scale matrix multiplication (GEMM).
50%UnverifiedIn large models based on the Transformer architecture, the vast majority of floating-point operations are concentrated in matrix multiplication, and the efficiency of GEMM kernels directly determines training throughput and inference latency.
50%UnverifiedFP8 has a narrower numerical range and lower precision, and correct usage requires fine-grained scaling factor management and numerical stability guarantees at the GEMM kernel level.
50%UnverifiedTileLang's most direct application scenarios are concentrated on core deep learning operators such as FlashAttention, GEMM (General Matrix Multiplication), and quantization and dequantization.
50%UnverifiedThe inference bottlenecks of MoE models lie mainly in two areas: KV Cache read/write (constrained by memory bandwidth) and GEMM computation for MoE-routed experts (a compute-intensive task with irregular shapes).
50%UnverifiedGeneral Matrix Multiplication (GEMM) is the most fundamental and computationally intensive operation in deep learning inference, and NVIDIA Tensor Cores and Google TPU systolic arrays are both specialized optimizations targeting GEMM.
50%UnverifiedThe experiment was conducted on a nine-core FP16 GEMM (half-precision general matrix multiplication) test set.
50%