ParaTTT
A parallelization method for test-time training (TTT) layers. By iteratively refining estimates of the inner-loop gradient trajectory, it parallelizes nonlinear memory of arbitrary depth while preserving the convergence properties of exact online gradient descent.
Timeline (last 90 days)
ParaTTT (PTTT) parallelizes non-linear memory of arbitrary depth by iteratively refining estimates of the inner-loop gradient trajectory.
Each refinement in ParaTTT comprises three chunk-parallel processes: a forward pass that evaluates the feature trajectory; an exact gated-delta-rule process that computes the readout trajectory and its feature gradients; and a backward pass that propagates these gradients into a new estimate.
The exact online gradient descent loop is the fixed point of ParaTTT's iterations, while decoding requires no refinement iterations.
The authors prove that the two-layer variant has geometric convergence under a sufficient contraction condition.
In practice, two iterations suffice.
At a 4K context, PTTT with two refinement steps evaluates the inner-loop trajectory at matched error 32 times faster than TTT-MLP using chunked approximation.
The full 340M model trains 4.9 times faster.
Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 on language modeling perplexity and zero-shot accuracy, remain competitive with Transformers of the same size, and have better recall capability than linear recurrence.
All Facts (8)
ParaTTT (PTTT) parallelizes non-linear memory of arbitrary depth by iteratively refining estimates of the inner-loop gradient trajectory.
50%UnverifiedEach refinement in ParaTTT comprises three chunk-parallel processes: a forward pass that evaluates the feature trajectory; an exact gated-delta-rule process that computes the readout trajectory and its feature gradients; and a backward pass that propagates these gradients into a new estimate.
50%UnverifiedThe exact online gradient descent loop is the fixed point of ParaTTT's iterations, while decoding requires no refinement iterations.
50%UnverifiedThe authors prove that the two-layer variant has geometric convergence under a sufficient contraction condition.
50%UnverifiedIn practice, two iterations suffice.
50%UnverifiedAt a 4K context, PTTT with two refinement steps evaluates the inner-loop trajectory at matched error 32 times faster than TTT-MLP using chunked approximation.
50%UnverifiedThe full 340M model trains 4.9 times faster.
50%UnverifiedModels trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 on language modeling perplexity and zero-shot accuracy, remain competitive with Transformers of the same size, and have better recall capability than linear recurrence.
50%