ZIP-SR
A 4-bit AdamW optimizer state quantization method that retains zero in the second-moment codebook and computes stochastic rounding probabilities in the preconditioner space.
Timeline (last 90 days)
The paper's proposed ZIP-SR method retains zero in the second-moment codebook and performs the probability computation for stochastic rounding in the preconditioner space.
Both ZIP-SR and ZE-EDEN use 4-bit NormalFloat (NF4) for the first moment and apply targeted stochastic rounding to the first moment of the LM-head during the last 10% of training.
Compared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.
ZIP-SR and ZE-EDEN make opposite choices regarding 'whether the codebook contains zero' yet both improve results, indicating that the key is not zero itself but whether explicit compensation is made for preconditioner distortion.
All Facts (4)
The paper's proposed ZIP-SR method retains zero in the second-moment codebook and performs the probability computation for stochastic rounding in the preconditioner space.
50%UnverifiedBoth ZIP-SR and ZE-EDEN use 4-bit NormalFloat (NF4) for the first moment and apply targeted stochastic rounding to the first moment of the LM-head during the last 10% of training.
50%UnverifiedCompared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.
50%UnverifiedZIP-SR and ZE-EDEN make opposite choices regarding 'whether the codebook contains zero' yet both improve results, indicating that the key is not zero itself but whether explicit compensation is made for preconditioner distortion.
50%