ReWY
An algebraic reconstruction optimization method for backpropagation in the Delta-rule family of linear attention operators. It accelerates gradient computation by eliminating the inverse adjoint and reducing the number of matrix multiplications associated with the WY matrix representation. The method applies to models such as DeltaNet, GDN, KDA and GDN2.
Timeline (last 90 days)
ReWY uses algebraic restructuring to compress the seven local matrix multiplications related to the WY matrix into three, and eliminates the inverse adjoint.
Under exact arithmetic, the ReWY method keeps the recurrence relations and gradients unchanged, without altering the model architecture or training objective.
ReWY is applicable to all four operators: DeltaNet, GDN, KDA and GDN2.
On NVIDIA GPUs, the geometric mean speedup of the full low-level backward pass is 1.15–1.81×.
On H100 and H200, the speedups of the four operators range from 1.18× to 1.70×, while retaining the original FP32 working matrices.
On Ascend 950DT, an independent public autograd benchmark (including gradient conversion) achieves speedups of 1.09–2.89×.
ReWY replaces pairwise vector gating with centralized matrix contraction.
The scalar gating reduction in ReWY no longer constructs the Gram matrix of keys, but instead reuses the key-gradient products.
The 1.15–1.81× speedups are geometric means of the low-level backward pass; end-to-end gains for the full model will vary with model scale and training configuration.
All Facts (9)
ReWY uses algebraic restructuring to compress the seven local matrix multiplications related to the WY matrix into three, and eliminates the inverse adjoint.
50%UnverifiedUnder exact arithmetic, the ReWY method keeps the recurrence relations and gradients unchanged, without altering the model architecture or training objective.
50%UnverifiedReWY is applicable to all four operators: DeltaNet, GDN, KDA and GDN2.
50%UnverifiedOn NVIDIA GPUs, the geometric mean speedup of the full low-level backward pass is 1.15–1.81×.
50%UnverifiedOn H100 and H200, the speedups of the four operators range from 1.18× to 1.70×, while retaining the original FP32 working matrices.
50%UnverifiedOn Ascend 950DT, an independent public autograd benchmark (including gradient conversion) achieves speedups of 1.09–2.89×.
50%UnverifiedReWY replaces pairwise vector gating with centralized matrix contraction.
50%UnverifiedThe scalar gating reduction in ReWY no longer constructs the Gram matrix of keys, but instead reuses the key-gradient products.
50%UnverifiedThe 1.15–1.81× speedups are geometric means of the low-level backward pass; end-to-end gains for the full model will vary with model scale and training configuration.
50%