ZE-EDEN
A 4-bit AdamW optimizer state quantization method that uses a second-moment codebook excluding zero and rescales the quantized second-moment blocks to mitigate preconditioner distortion.
Timeline (last 90 days)
The paper's proposed ZE-EDEN method uses a second-moment codebook that excludes zero and rescales the quantized second-moment blocks to mitigate the preconditioner distortion caused by the positive quantization lower bound.
Both ZIP-SR and ZE-EDEN use 4-bit NormalFloat (NF4) for the first moment and apply targeted stochastic rounding to the first moment of the LM-head during the last 10% of training.
Compared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.
ZIP-SR and ZE-EDEN make opposite choices regarding 'whether the codebook contains zero' yet both improve results, indicating that the key is not zero itself but whether explicit compensation is made for preconditioner distortion.
All Facts (4)
The paper's proposed ZE-EDEN method uses a second-moment codebook that excludes zero and rescales the quantized second-moment blocks to mitigate the preconditioner distortion caused by the positive quantization lower bound.
50%UnverifiedBoth ZIP-SR and ZE-EDEN use 4-bit NormalFloat (NF4) for the first moment and apply targeted stochastic rounding to the first moment of the LM-head during the last 10% of training.
50%UnverifiedCompared with TorchAO's 4-bit AdamW, ZIP-SR and ZE-EDEN reduce the average validation loss gap relative to 32-bit AdamW at every evaluated scale, with a maximum gap reduction of 70%.
50%UnverifiedZIP-SR and ZE-EDEN make opposite choices regarding 'whether the codebook contains zero' yet both improve results, indicating that the key is not zero itself but whether explicit compensation is made for preconditioner distortion.
50%